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OPERATIONAL  ASSESSMENT  OE  TOOLS  EOR  ACCELERATING  LEADER 
DEVELOPMENT  (ALD):  VOLUME  I,  CAPSTONE  REPORT 

EXECUTIVE  SUMMARY 


Research  Requirement: 

To  support  Army  transformation  efforts,  the  U.S.  Army  Research  Institute’s  Eort 
Leavenworth  Research  Unit  (ELRU)  conducts  research  and  development  to  advance  the  state  of 
the  art  for  building  excellence  among  Army  leaders.  Today’s  operational  environment  makes  it 
very  difficult  for  leaders  to  pursue  robust  professional  development.  Yet  leaders  are  expected  to 
master  more  skills  earlier  in  their  careers  than  ever  before.  To  meet  the  need,  ELRU  researchers 
have  developed  an  innovative  toolkit  under  a  program  known  as  ALD — Accelerating  Leader 
Development.  The  final  phase  of  the  program  entailed  a  field  assessment  of  the  toolkit. 

Procedure: 

The  research  team  installed  the  toolkit  on  a  Web  site  server  with  learning  management 
and  database  capabilities.  Eollowing  a  pilot  test,  the  team  recruited  tactical  units  and  user  juries 
to  provide  feedback  on  the  tools.  Two  combat  battalions  (142  commissioned  officers  and  non¬ 
commissioned  officers)  participated  in  the  operational  test.  In  addition,  user  jury  sessions  at  six 
different  installations  drew  140  company  grade  officers  and  non-commissioned  officers.  Data 
collection  methods  included  Web  site  logging,  pre-  and  post-testing  (online),  embedded  testing 
(online),  surveys  (online  and  hardcopy),  and  interviews.  The  resulting  data  were  analyzed  to 
assess  the  suitability,  acceptability,  effectiveness,  and  impact  of  the  toolkit. 

Eindings: 

Of  the  unit  participants,  62  worked  with  one  or  more  online  ALD  tools  for  varying 
amounts  of  time,  while  more  than  90  took  part  in  offline  group  training  with  filmed  vignettes. 

The  pressures  of  the  operational  environment  seriously  limited  the  time  available  to  work  with 
the  tools.  The  user  satisfaction  results  indicated  the  toolkit  provides  a  reasonable  mix  of 
diagnostic  and  training  capabilities,  but  completeness  remains  an  open  question.  The  tools  appear 
to  be  more  suitable  for  use  in  pre-commissioning  and  military  education  programs.  The  various 
modules  offer  moderately  favorable  acceptability,  learning  effectiveness,  and  value.  However, 
the  toolkit’s  online  learning  methods  may  need  expanded  capabilities  for  interpersonal 
interaction  with  mentors  and  peers  to  become  fully  suitable  for  leader  self-development. 

Utilization  and  Dissemination  of  Eindings: 

The  findings  and  lessons  learned  from  this  research  (including  the  appendices  in  Volume 
II)  can  help  decision  makers,  program  architects,  training  developers,  and  research  investigators 
enhance  self-development  programs  for  the  Army’s  junior  leaders.  Unit  trainers  and  training 
managers  can  use  the  toolkit  to  expand  their  leader  development  programs.  The  ultimate  payoff 
will  come  from  leveraging  stronger  leader  competencies  to  enhance  mission  performance. 
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Introduction 

As  the  Army  transforms  to  a  modular  structure,  the  need  for  high-quality  leadership 
throughout  the  force  is  attracting  more  and  more  attention.  The  contemporary  operational 
environment  (COE)  demands  excellence  among  tactical  leaders,  yet  compelling  pressures  make 
it  more  challenging  than  ever  for  leaders  to  pursue  professional  development.  The  challenges 
were  recognized  several  years  ago  by  the  Army  Training  and  Leader  Development  Panel  (U.S. 
Department  of  the  Army,  2001).  The  panel’s  recommendations  included  focusing  on  self- 
awareness  and  adaptability,  exploiting  network  technology,  dedicating  resources  to  distributed 
learning,  and  introducing  a  multi-source  feedback  strategy.  Notably,  the  distributed  learning 
recommendation  focused  on  providing  “lieutenants  access  to  Web-based,  self-development 
modules  to  accelerate  and  enhance  their  ability  to  meet  the  requirements  of  [captain-level] 
assignments”  (U.S.  Department  of  the  Army,  2001,  p.  OS-19).  Since  publication  of  the  panel’s 
report,  the  operational  tempo  (OPTEMPO)  has  increased  and  modularity  has  become  a  major 
initiative.  These  factors  have  further  complicated  the  leader  development  challenges  facing 
Army  officers.  Innovative  training  approaches  are  needed  to  enable  young  officers  to  build  their 
leadership  skills  faster  and  assume  higher  levels  of  responsibility  earlier  in  their  careers. 

Responding  to  the  emerging  imperatives,  the  U.S.  Army  Research  Institute  (ARI) 
initiated  a  program  entitled  “Accelerating  Leader  Development”  (ALD).  Begun  in  LY04  under 
an  Army  Technology  Objective  (ATO),  the  program  aimed  to  develop  approaches,  methods  and 
tools  to  enhance  the  self-development  of  company-grade  officers.  The  work  supported  the  U.S. 
Army  Command  and  General  Staff  College,  the  Center  for  Army  Leadership,  and  the  School  for 
Command  Preparation.  As  the  lead  organization  for  the  ALD  program,  ARI’s  Lort  Leavenworth 
Research  Unit  (LLRU)  defined  the  leadership  skills  that  are  critical  in  today’s  operational 
environment.  Investigators  proceeded  to  develop  a  family  of  self-development  tools  for  building 
the  critical  skills.  The  culminating  phase  of  the  ATO  entailed  an  operational  demonstration  and 
assessment  of  the  ALD  tools  conducted  among  the  target  audience. 

This  document — Volume  I — presents  the  methods  and  findings  of  the  operational 
demonstration  and  assessment  of  the  ALD  toolkit,  with  emphasis  on  training  methodology.  It 
offers  valuable  information  for  decision  authorities,  program  architects,  training  developers,  and 
researchers  working  to  advance  the  Army’s  leader  development  capabilities.  Volume  II  contains 
the  appendices.  A  companion  report  describes  the  technology  aspects  of  the  toolkit  and  the 
delivery  environment  (Woller,  Garven,  &  Leibrecht,  in  preparation). 

Background 

The  overarching  goal  of  the  ATO  was  to  establish  an  innovative,  integrated  toolkit  for 
building  leadership  competencies  to  meet  the  demands  of  the  COE.  As  seen  in  Table  1,  the 
program  focused  on  competencies  that  contribute  significantly  to  effective  leadership — self- 
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awareness,  metaeognition,  eritical  thinking  skills,  and  interpersonal  skills.  These  dimensions 
resulted  from  systematie  literature  review  and  researeh  (e.g..  Day,  Zaeearo,  &  Halpin,  2004). 


Table  1 

Critical  Leader  Competencies 


Competency 

Description 

Self-awareness 

Understanding  of  one’s  own  strengths,  weaknesses,  aptitudes,  and  attributes 

Metacognition 

Capacity  to  monitor  and  shape  own  self-development  approaches  and  attitudes 

Critical  Thinking 

Ability  to  apply  structured  problem-solving  techniques  to  leadership  challenges 

Interpersonal  Skills 

Ability  to  handle  person-to-person  and  cultural  challenges  as  a  unit  leader 

Seientists  in  ARI  have  been  investigating  and  building  leader  development  solutions  for  a 
deeade  or  more.  Much  of  the  initial  work  focused  on  research  frameworks  and  dimensions 
offering  promise  for  advancing  the  state  of  the  art  (Zaeearo,  Klimoski,  &  Gade,  1999;  Boyce, 
Gade,  Zaeearo,  &  Klimoski,  2000;  Day  &  Halpin,  2001).  Acknowledging  the  importance  of  self- 
awareness  in  leader  development,  ARI  scientists  established  approaches  and  techniques  for  self- 
assessment  (Bass  &  Avolio,  1990,  1995;  Zaeearo,  Klimoski,  Boyce,  et  ah,  1999;  Cortina  et  ah, 
2004)  and  multi-source  assessment  (e.g.,  Halpin,  1997;  Karrasch,  Halpin,  &  Keene,  1997; 
Karrasch  &  Halpin,  1999).  These  efforts  led  to  the  development  of  various  instruments  for 
assessing  one’s  own  leadership  experiences,  attitudes,  preferences,  processes  and  outcomes. 

Other  investigators  explored  the  role  of  tacit  knowledge  in  leadership  (Horvath,  Hedlund, 
Snook,  Forsythe,  &  Sternberg,  1998;  Sternberg  et  ah,  1999;  Hedlund,  Sternberg,  &  Psotka,  2000; 
Antonakis,  Hedlund,  Pretz,  &  Sternberg,  2002).  Subsequent  work  extended  into  methods  for 
training  tacit  knowledge  (Cianciolo,  Antonakis,  &  Sternberg,  2001;  Matthew,  Cianciolo,  & 
Sternberg,  2005)  and  techniques  for  assessing  it  (Hedlund  et  ah,  2003).  Cianciolo,  Antonakis, 
and  Sternberg  (2004)  went  on  to  expand  the  tacit  knowledge  concepts  into  a  broader  formulation 
of  practical  intelligence  and  leadership. 

In  parallel  research,  scientists  analyzed  the  role  of  critical  thinking  in  military  leadership 
and  created  the  foundation  for  training  critical  thinking  skills  among  Army  leaders  (Cohen, 
Thompson,  Adelman,  Bresnick,  &  Riedel,  1999;  Riedel,  Morath,  &  McGonigle,  2001).  This  led 
to  development  of  methods  and  tools  for  enriching  critical  thinking  abilities  (Lynch  &  Wolcott, 
2001;  Fischer,  Spiker,  &  Riedel,  2008a,  2008b,  2009).  These  tools  are  seen  as  especially 
important  in  light  of  the  Army’s  emergent  emphasis  on  adaptive  leaders  who  are  mentally  agile 
and  skilled  in  solving  poorly  structured  problems  (e.g.,  U.S.  Department  of  the  Army,  2006). 

Still  other  researchers  explored  the  dimensions  of  interpersonal  skills  and  their  role  in 
leadership  (Carpenter  &  Wisecarver,  2004;  Zbylut  &  Ward,  2004a).  Their  work  set  the  stage  for 
developing  film-based  vignettes  that  challenge  leaders  to  expand  their  understanding  of  social 
dynamics  in  realistic  scenarios  (Hill,  Douglas,  Gordon,  Pighin,  &  van  Velsen,  2003;  Hill, 
Gordon,  &  Kim,  2004;  Zbylut  &  Ward,  2004b;  Zbylut,  Ward,  &  Mark,  2005;  Nobel  et  ah,  2006; 
Zbylut  et  ah,  2007;  Zbylut,  Brunner,  Vowels,  &  Kim,  2007).  The  vignettes  utilize  facilitated 
group  discussion  and  multimedia  mentoring  to  interactively  enhance  the  learning  experience. 
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Some  of  the  ARI  researeh  has  addressed  the  theoretieal  underpinnings  of  leadership 
development.  Avolio  and  Yammarino  (2003)  examined  leadership  eonstruets  derived  from 
transformational  leadership  theory  (e.g.,  Bass,  1996),  as  distinguished  from  transaetional 
leadership  theory.  In  a  longitudinal  field  study  they  demonstrated  that  the  impaet  of  a 
transformational  leadership  intervention  eould  be  measured  in  terms  of  leaders’  attitudes, 
applieation  aetivities,  personal  development,  and  unit  outeomes.  Boyee,  Wiseearver,  and  Zaeearo 
(2005)  ereated  a  struetural  model  of  leader  self-development  and  demonstrated  that  the  multi¬ 
dimensional  model  provided  a  unifying  framework  for  understanding  the  role  of  individual 
eharaeteristies.  In  addition.  Carpenter  and  Wiseearver  (2004)  eonstrueted  a  model  of  dimensions 
defining  interpersonal  performanee  in  the  form  of  a  behavioral  taxonomy.  They  eondueted 
empirieal  researeh  that  eonfirmed  the  suitability  of  the  proposed  model. 

The  work  eondueted  under  the  ATO  and  predeeessor  efforts  yielded  tools  in  three  high- 
priority  areas — self-awareness/metaeognition,  eritieal  thinking  skills,  and  interpersonal  skills. 

The  prototype  tools  inelude  (a)  self-assessment  instruments  plus  self-guided  learning  aetivities  to 
foster  metaeognitive  insights  into  one’s  own  leader  behaviors,  (b)  online  instruetional  paekages 
designed  to  build  eritieal  thinking  skills,  and  (e)  film-based  vignettes  that  hone  interpersonal 
skills  via  group  diseussion.  The  training  program  harnesses  new  approaehes  that  inelude  self- 
assessment  tools,  struetured  opportunities  to  link  experiential  and  formal  knowledge,  immersive 
ehallenges  for  eritieal  thinking,  story  telling,  and  eomputerized  eoaehing  and  feedbaek.  The  ALD 
toolkit,  deseribed  in  the  next  seetion,  aims  to  boost  the  Army’s  ability  to  develop  leaders  who 
ean  meet  the  ehallenges  and  rigors  of  the  COE.  By  enhaneing  leader  development  proeesses  and 
outeomes,  the  toolkit  is  designed  to  build  high-performing  leader  teams.  The  payoff  is  expeeted 
to  direetly  faeilitate  Army  transformation  efforts. 

The  ATO  eulminated  in  a  field-oriented  demonstration  and  assessment  using  U.S.  Army 
Soldiers  to  represent  the  primary  target  audienee.  The  overarehing  goal  of  this  final  phase  was 
two-fold:  (a)  to  demonstrate  operational  utilization  of  the  ALD  produets  and  (b)  to  assess  their 
aeeeptability  and  potential  eontributions  in  the  hands  of  users.  The  operational  demonstration 
eonstituted  a  eapstone  integrated  teehnology  demonstration  that  put  the  family  of  multi¬ 
dimensional  produets  to  an  aeid  test.  In  designing  the  demonstration  and  assessment,  ARI 
seientists  defined  multiple  opportunities  and  meehanisms  to  obtain  meaningful  user  feedbaek. 

Problem  Definition 

A  number  of  faetors  in  today’s  national  defense  environment  impaet  leadership 
development  among  eompany  grade  offieers.  The  most  signifieant  faetors  are: 

•  The  COE,  espeeially  as  impaeted  by  operational  variables  (politieal,  military, 
eeonomie,  soeial,  information,  and  infrastrueture — PMESII)  and  unified  aetion. 

•  The  Army  Eoree  Generation  (AREORGEN)  model  with  the  attendant  transition  to  the 
modular  foree,  modular  organization  realignments,  and  unit  lifeeyele. 

•  The  persistently  elevated  OPTEMPO  highlighted  by  deployment  requirements  that 
stress  units  and  Soldiers. 

Military  operations  in  the  COE  require  inereasingly  preeise  eoordination  of  dispersed 
forees  within  a  rapidly  ehanging  deployment  environment.  Army  leaders  faee  greater  ambiguity 
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and  complexity  across  the  full  spectrum  of  military  operations,  and  these  conditions  are  expected 
to  continue.  This  environment,  coupled  with  a  wide  variety  of  missions,  demands  that  leaders  at 
all  levels  be  adaptive,  self-aware,  and  able  to  perform  under  complex  conditions  of  uncertainty. 
The  Army’s  leaders  need  to  promote  human  capital  and  knowledge,  develop  teams,  resist  stress, 
and  adapt  quickly  and  effectively  in  volatile  and  unstructured  environments.  They  must  be  able 
to  think  critically  and  creatively  and  to  shift  their  perspective  as  the  situation  demands. 

The  cascading  PMESII  consequences  of  current  tactical  missions  not  only  require  Army 
leaders  to  deal  with  complex  and  uncertain  conditions;  they  demand  that  company  grade  officers 
act  independently  at  levels  beyond  those  previously  expected  of  their  grade  and  time  in  service. 
Company  grade  officers  must  be  trained  to  lead  and  act  in  an  ever  changing  tactical  environment 
and  to  do  so  at  an  increasing  level  of  independence. 

The  expanding  role  of  unified  action  is  a  major  factor  in  the  COE’s  complexity.  This 
dimension  of  contemporary  operations  combines  joint,  interagency,  intergovernmental,  and 
multinational  elements.  Joint  and  unified  military  operations  require  leaders  to  coordinate 
dispersed  forces  within  an  information  warfare  environment  that  can  change  quickly.  Eeadership 
in  this  age  of  mushrooming  knowledge,  networks,  and  dynamic  information  environments 
requires  new  knowledge  and  skills — cognitive,  interpersonal,  and  systems-specific. 

Today’s  high-OPTEMPO  context  of  military  operations  renders  obsolete  the  old  Army 
readiness  paradigm  of  alert-train-deploy.  Continuous,  full-spectrum  expeditionary  operations  are 
the  new  reality.  To  meet  the  new  requirements,  the  Army  has  developed  a  force  generation 
process — AREORGEN  (Eigure  1) — to  provide  combatant  commanders  and  civil  authorities  with 
rapidly  deployable,  employable,  and  sustainable  force  capabilities  packages  tailored  to  specific 
mission  requirements  (McNeil,  2005).  Implementation  of  the  three-phase  AREORGEN  process 
cuts  across  the  entire  Army  and  profoundly  impacts  units  and  their  leaders.  In  a  sweeping 
transformation  involving  changes  to  doctrine  and  organization,  the  Army  is  transitioning  from  a 
division-based  force  to  a  modular  force  based  on  brigade  combat  teams  (BCTs).  The  modular 
force  structure  aims  to  meet  joint  and  coalition  needs  for  responsive,  flexible,  rapid,  worldwide 
deployment.  In  the  process,  the  emphasis  is  shifting  from  deliberate  to  adaptive  war  planning. 

To  support  this  transformation,  the  Army  leadership  has  revamped  the  processes  for 
training  tactical  units.  The  new  unit  lifecycle  training  model  (McNeil,  2005)  is  geared  to  leverage 
individual,  team  and  collective  training,  including  an  externally  evaluated  certifying  event.  The 
goal  is  to  produce  cohesive  and  deployment-ready  units  with  well  trained  Soldiers  led  by 
confident  leaders.  All  phases  of  the  lifecycle  are  intensive  and  place  great  demands  on  the 
company  grade  officers’  time,  limiting  opportunities  for  professional  development  beyond  what 
occurs  in  the  execution  of  their  routine  duties.  Yet  preparing  company  grade  leaders  to  be  more 
effective  and  adaptive  is  important  in  meeting  the  challenges  faced  by  the  Army. 

Another  component  of  the  Army’s  modular  transition  is  organizational  realignment — 
moving  organizations  and  Soldiers  to  assignments  at  new  installations  to  align  with  their  new 
modular  force  role.  The  alignment  actions  add  to  the  mix  of  requirements  claiming  the  time  and 
energies  of  company  grade  leaders.  The  multi-dimensional  aspects  of  modular  transformation. 
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combined  with  the  pressures  of  the  COE,  form  the  backdrop  for  junior  leader  development 
ehallenges  faeing  the  Army  today. 
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Figure  1.  The  ARFORGEN  proeess  (taken  from  U.S.  Army  Forees  Command,  2008). 

As  an  indieator  of  the  pace  of  operations,  OPTEMPO  is  an  overarching  contributor  to  the 
issues  eonfronting  today’s  Army  leaders  and  the  development  of  eompany  grade  leadership 
skills.  The  OPTEMPO  is  not  expeeted  to  ease  in  the  foreseeable  future.  Historieally,  there  has 
always  been  a  surplus  of  demands  on  the  eompany  grade  offieer’s  time.  Extended  deployment 
tours  eoupled  with  the  imperative  of  training  to  eombat  readiness  for  the  next  deployment  have 
greatly  inereased  these  demands  and  severely  constrained  the  time  available  for  professional 
development.  In  parallel  the  COE  demands  have  inereased,  expanding  the  need  for  professional 
development  of  eompany  grade  offieers. 

As  a  learning  proeess,  leader  development  shapes  knowledge  and  skills  (cognitive, 
interpersonal,  ete.)  as  well  as  attitudes  and  values.  Traditionally,  offieer  development  has  mainly 
foeused  on  aequiring  knowledge.  The  knowledge,  skills,  and  wisdom  required  for  effeetive 
leadership  have  eome  through  years  of  experience  in  a  broad  range  of  settings.  However,  the 
COE  demands  expanded  leadership  competeneies  from  relatively  junior  leaders.  New  methods 
and  tools  are  required  to  meet  the  leader  development  needs  of  the  Army’s  eompany  grade 
offieers  if  they  are  to  be  ready  to  perform  in  the  COE. 

Technical  Objectives 

The  impetus  for  the  ALD  ATO  originated  with  the  interaetion  of  two  eritieal  and 
conflieting  factors — the  COE’s  inereasing  demands  and  the  decreasing  time  available  for 
professional  development  of  eompany  grade  offieers.  The  combination  of  these  two  factors 
drove  the  researeh  and  development  program  that  ereated  innovative  leadership  development 
tools  for  eompany  grade  leaders.  Demonstrating  and  assessing  the  effeetiveness  of  the  new  tools 
was  an  essential  step  in  erafting  recommendations  for  refining  and  implementing  them. 
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As  the  culminating  phase  of  the  ATO,  the  present  research  pursued  two  primary  goals — 
demonstrate  the  operational  utilization  of  the  ALD  toolkit,  and  assess  its  contributions  and  value 
to  the  Army’s  leader  development  efforts.  To  align  the  results  with  the  needs  of  tactical  units,  an 
operational  demonstration/assessment  was  deemed  imperative. 

The  following  technical  objectives  structured  the  planning,  preparation  and  execution  of 
the  demonstration  and  assessment  of  the  ALD  toolkit: 

♦  Conduct  an  integrated  technology  demonstration  of  the  ALD  products 

♦  Measure  the  acceptability,  utilization,  and  effectiveness  of  individual  ALD  products 

♦  Examine  the  impact  of  the  ALD  products  on  leader  competencies 

♦  Quantify  the  effects  of  ALD  training  on  unit  climate  and  cohesion 

♦  Explore  the  ALD  program’s  impact  on  unit  training  and  performance 

♦  Determine  the  completeness  and  sufficiency  of  the  ALD  program 

♦  Capture  good  ideas  for  improving  the  ALD  products  and  program 


The  Leader  Development  Toolkit 

The  ALD  toolkit  resulted  from  a  series  of  research  and  technology  transfer  projects 
initiated  at  the  outset  of  the  ATO.  As  a  family  of  innovative  leader  development  tools,  the  ALD 
toolkit  set  the  stage  for  the  operational  demonstration  and  assessment  conducted  in  the  current 
project.  This  section  describes  the  components  of  the  toolkit  that  defined  the  ALD  training 
program,  including  the  training  support  tools  and  the  utilization  model. 

Overview 

The  ALD  toolkit  brings  a  new  approach  to  leader  development  training  that  exploits  a 
variety  of  techniques  embedded  in  an  overall  program  of  self-assessment  and  self-development 
capabilities  (Eigure  2).  The  program  provides  Web-accessed  tools  (with  one  exception)  and  a 
recommended  model  for  harnessing  them.  The  tools  are  intended  for  use  by  company  grade 
officers  in  operational  units,  regardless  of  duty  position.  The  fundamental  intent  is  to  increase 
leadership  skills  by  helping  officers  to  become  self-aware,  adaptive  and  agile  decision  makers. 
The  toolkit  contains  two  types  of  tools — diagnostic  and  training — that  support  self-paced 
assessment  and  learning. 


Diagnostic  Tools 

The  diagnostic  tools  were  selected  with  two  global  functions  in  mind — self-assessment 
and  feedback,  and  measurement  of  training  efficacy.  The  self-assessment  function  was  intended 
to  (a)  provide  an  objective  basis  for  self-awareness  and  deliberate  planning  of  self-development 
activities  and  (b)  “give  back”  to  the  Soldiers  who  participated.  One  of  the  main  complaints 
Soldiers  express  about  “surveys”  is  that  they  rarefy  receive  feedback,  and  the  diagnostic  tools 
addressed  the  issue.  The  second  global  function — training  efficacy — aimed  to  quantify  changes 
in  leadership  knowledge  and  skills  that  might  result  from  using  the  ALD  toolkit. 
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Figure  2.  ALD  program  model  of  leader  growth. 

Table  2  lists  the  seven  instruments  included  as  online  diagnostic  tools.  Four  of  them 
served  as  both  pre-  and  post-training  measures  because  of  their  expected  sensitivity  to  ALD 
learning  effects.  Two  of  the  instruments  (Multifactor  Leadership  Questionnaire  and  Tacit 
Knowledge  for  Military  Leaders  instrument)  were  configured  in  alternate  versions  to  avoid 
carryover  effects  from  the  pre-  to  post-training  measurement.  The  other  dual-role  (pre  and  post) 
tools  were  single-version  because  they  posed  negligible  risk  of  carryover  effects.  In  some  cases 
only  a  portion  of  the  instrument’s  measures  or  subscales  were  used  in  this  research.  All  of  the 
diagnostic  tools  were  regarded  as  candidates  to  support  analysis  of  individual  differences. 


Table  2 

Diagnostic  Tools  of  the  ALD  Toolkit 


Tool 

Role 

Purpose 

Demographic  Survey 

Pre  only 

Capture  biographical  and  demographic  data 

Personality  Assessment 

Pre  only 

Measure  stable  traits  to  boost  personal  insights 

Leadership  Experiences  Survey 

Pre  only 

Record  history  of  leadership  assignments 

Multifactor  Leadership  Questionnaire  (MLQ) 

Pre  &  Post 

Assess  transformational/transactional  styles 

Tacit  Knowledge  for  Military  Leadership  (TKML) 

Pre  &  Post 

Document  tacit  leadership  knowledge 

Leadership  Opinions  Survey 

Pre  &  Post 

Measure  self-development  interest/motivation 

Unit  Cohesion  Index  (UCI) 

Pre  &  Post 

Measure  cohesion  within  own  unit 
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Demographic  Survey 

The  demographic  survey  was  developed  by  the  research  team  especially  for  the  ALD 
demonstration  and  assessment.  It  queries  participants  regarding  individual  variables  and  military 
background.  The  query  dimensions  include  gender,  age,  ethnicity,  native  language,  time  on 
active  duty,  prior  service,  rank,  branch  of  service,  military  training,  and  deployment  experience. 
The  tool  requires  less  than  15  min  to  complete. 

Personality  Assessment 

The  International  Personality  Item  Pool  inventory  (Goldberg,  1999)  is  used  to  measure 
personality  traits.  The  short  form  version  of  this  instrument  is  intended  to  inform  participants  of 
key  personality  facets  that  may  contribute  to  their  leadership  style.  The  inventory  categorizes 
personality  along  five  factors — extraversion,  agreeableness,  conscientiousness,  neuroticism,  and 
openness  to  experience.  Within  each  factor  are  multiple  subscales.  Completing  the  inventory 
yields  a  narrative  report  to  the  participant  describing  his/her  standing  on  each  factor  and  subscale 
in  qualitative  terms  such  as  low  anger  or  average  assertiveness.  For  this  tool,  participants  were 
hyperlinked  to  another  Web  site  (http://www.personal.psu.edu/facultv/i/5/i5i/IPIP/).  Completing 
the  inventory  requires  about  45  min. 

Leadership  Experiences  Survey 

The  Leadership  Experiences  survey  is  taken  from  the  Leadership  Attitudes,  Opinions, 
and  Experiences  Survey  (Zaccaro,  2002).  The  instrument  entails  a  retrospective  look  at  specific 
leadership  situations  encountered  during  the  past  5  years,  such  as  working  with  people  from 
other  cultures.  Eor  each  situation  the  participant  selects  responses  on  three  scales:  (a)  number  of 
opportunities  to  engage  in  the  activity;  (b)  density  (number  of  times  the  activity  required  primary 
focus);  and  (c)  point  in  the  job  assignment  during  which  the  activity  was  engaged  in  the  most. 
The  survey  presents  31  situations  and  takes  about  30  min  to  complete. 

Multifactor  Leadership  Questionnaire  (MLQ) 

The  MEQ  instrument  (Bass  &  Avolio,  1995)  assesses  various  aspects  of  transformational 
and  transactional  leadership.  It  prompts  a  participant  to  describe  his/her  leadership  style  by  using 
a  5 -point  rating  scale  to  indicate  how  frequently  various  statements  fit — for  example,  “I  seek 
differing  perspectives  when  solving  problems.”  The  rating  scale  ranges  from  “not  at  all”  to 
“frequently.”  The  5X-short  version  contains  45  items,  responses  to  which  are  combined  into 
seven  subscales  used  to  differentiate  leadership  style.  The  MLQ  requires  about  30  min. 

Tacit  Knowledge  for  Military  Leadership  (TKML)  Survey 

The  TKML  (Hedlund  et  ah,  1999)  assesses  an  officer’s  knowledge  gained  through 
experience  as  opposed  to  formal  training.  Participants  read  typical  situations  encountered  by 
military  leaders,  then  rate  multiple  options  for  handling  a  situation  using  a  9-point  scale  ranging 
from  “extremely  bad”  to  “extremely  good.”  Each  participant’s  rating  of  an  option  is  compared 
with  experts’  ratings  to  generate  a  feedback  summary.  For  the  ALD  toolkit,  the  company  level 
version  of  the  TKML  contains  10  situations  (with  an  average  of  9  optional  actions  each),  while 
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the  platoon  version  presents  8  situations  (averaging  10  optional  actions  each).  A  given 
participant  completes  only  one  version,  based  on  rank  and  position  specified  in  the  Demographic 
Survey.  Completing  either  version  requires  up  to  1  hr. 

Leadership  Opinions  Survey 

The  Leadership  Opinions  survey  also  comes  from  the  Leadership  Attitudes,  Opinions, 
and  Experiences  Survey  (Zaccaro,  2002).  This  instrument  asks  participants  to  indicate  their 
opinions  and  attitudes  about  various  aspects  related  to  leadership  style,  such  as  “To  what  extent 
are  you  able  to  size  up  another  person  quickly?”  The  instrument  presents  41  questions  with  5- 
point  Likert-type  response  scales,  taking  about  30  min  to  complete. 

Unit  Cohesion  Index  (UCI) 

The  UCI  (Siebold  &  Kelly,  1988)  was  used  to  provide  an  overall  look  at  the  participants’ 
perceptions  of  unit  cohesiveness  based  on  social  dynamic  factors  including  trust  and  teamwork. 
Responses  from  the  members  of  a  unit  can  be  pooled  to  produce  a  collective  measure.  The  tool 
contains  24  statements  such  as  “Members  trust  each  other  in  this  unit,”  with  participants  rating 
their  agreement/disagreement  using  a  5 -point  scale  ranging  from  “strongly  agree”  to  “strongly 
disagree.”  The  rating  scale  includes  an  “unable  to  judge”  option.  The  responses  are  combined 
into  10  subscales  characterizing  various  aspects  of  social  cohesion.  This  tool  usually  takes  less 
than  15  min  to  complete. 


Training  Tools 

The  work  performed  under  the  ATO  produced  three  innovative  tools  for  delivering  self¬ 
development  training  to  company  grade  officers.  The  training  tools  were  designed  by  applying 
the  results  of  research  on  competencies  critical  to  effective  leadership  in  the  COE  (see  Table  1). 
The  ALD  training  tools  are  listed  in  Table  3  along  with  the  intended  purpose  of  each.  The 
following  paragraphs  describe  the  tools  and  their  components. 

Table  3 

Training  Tools  of  the  ALD  Toolkit 


Tool 

Mode 

Purpose 

Leader  Adaptability  Self-Training  System  (LASTS) 

Individual 

Sharpen  metacognitive  self-development  skills 

Critical  Thinking  Skills  (CTS)  Modules 

Individual 

Enhance  cognitive  abilities  critical  for  leaders 

Army  Excellence  in  Leadership  (AXL)  Vignettes 

Group 

Build  interpersonal  skills  via  situated  learning 

Leader  Adaptability  Self-Training  System  (LASTS) 

The  LASTS  tool  (Cortina  et  ah,  2004)  is  designed  to  promote  proactive,  deliberate  leader 
development  by  improving  an  officer’s  abilities  to  conceptualize  and  think  about  the  goals  and 
processes  involved  in  self-development.  The  tool  contains  five  modules: 
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•  Self-appraisal,  including  self-appraisal  biases,  identifying  desired  characteristics,  and 
defining  a  “possible  self” 

•  Self  and  social  awareness,  including  awareness  and  leadership,  self  awareness, 
managing  emotions,  and  social  awareness. 

•  Self  regulation  and  self  development,  including  goal  setting,  identifying  resources 
(mentors,  learning  partners,  etc.),  monitoring  progress,  and  addressing  discrepancies. 

•  Self  development  learning  contracts,  including  a  reflection  exercise,  a  self¬ 
development  goals  exercise,  and  developing  a  learning  contract. 

•  Application  exercise,  focusing  on  adaptive  problem  solving. 

Available  in  two  alternate  versions  (high  vs.  low  structure),  each  module  consists  of 
slides  (primarily  text)  that  the  participant  views  at  his/her  own  pace.  In  addition  to  the  primary 
training  modules,  the  LASTS  suite  includes  an  introduction  (overview)  and  a  38-item  survey 
designed  to  determine  whether  a  highly  structured  or  a  less  structured  (discovery  learning) 
training  approach  would  fit  the  participant  better.  Both  of  these  components  are  optional.  The 
complete  set  of  modules  requires  3-4  hr  to  complete. 

Critical  Thinking  Skills  (CTS) 

The  CTS  training  program  (Fischer,  et  al,  2008)  aims  to  improve  cognitive  skills  by 
enhancing  a  leader’s  ability  to  (a)  critically  evaluate  novel  situations,  (b)  develop  creative, 
effective  solutions,  and  (c)  make  sound  decisions  under  time-constrained  circumstances.  The 
training  was  designed  originally  for  individual  learning  in  an  academic  setting  but  was  modified 
to  better  fit  a  military  audience.  The  program  comprises  eight  interactive,  multimedia  modules, 
each  requiring  about  2  hr  to  complete. 

The  CTS  modules  are  set  in  a  framework  of  gathering,  processing,  and  evaluating  new 
information  to  meet  diverse  needs.  The  eight  modules  cover  the  following  action-oriented 
principles: 

•  Frame  the  message 

•  Recognize  the  gist  in  material 

•  Develop  an  explanation 

•  Generalize  from  specific  instances  to  broader  classes 

•  Use  mental  imagery  to  evaluate  plans 

•  Challenge  one’s  biases 

•  Examine  other  people’s  perspectives 

•  Decide  when  to  seek  information  based  on  its  value  and  cost 

Each  module  trains  three  to  six  sub-skills  and  includes  practical  exercises  and  test 
exercises  which  yield  scores  and  provide  feedback  to  the  participant.  In  addition  to  the  training 
lessons  there  are  pre-  and  post-tests  for  self-evaluation. 

As  the  operational  demonstration  got  underway,  the  research  team  decided  to  add  the 
CTS  post- tests  to  the  end-of-program  (“posf ’)  measures  described  earlier  (see  Table  2).  The 
intent  was  to  assess  the  retention  of  CTS  learning  following  the  end  of  AED  training.  Each 
stand-alone  post- test  was  expected  to  take  less  than  30  min. 
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Army  Excellence  in  Leadership  (AXL)  Vignettes 

The  AXL  component  of  the  toolkit  (Zbylut  &  Ward,  2004b;  Zbylut  et  ah,  2007)  deals 
with  the  interpersonal  aspects  of  leadership,  with  emphasis  on  realistic  complexities  of  the  COE. 
The  tool  contains  two  vignettes,  titled  Power  Hungry  and  Tripwire,  which  are  based  on  Soldiers’ 
anecdotes  from  operations  in  Afghanistan  and  Iraq.  Geared  to  catalyze  the  acquisition  of  tacit 
knowledge,  the  vignettes  are  filmed  case  studies  combining  storytelling  elements  with  situated 
training  techniques. 

The  AXL  vignettes  are  designed  to  be  used  interactively  in  small  group  settings  with  a 
knowledgeable  facilitator  present.  They  are  internet  downloadable  and  adaptable  for  individual 
training,  but  they  are  best  used  in  conjunction  with  small  group  discussions  that  focus  on  a 
specific  teaching  theme.  Guidelines  are  available  to  help  orchestrate  the  discussion.  Group 
members  have  a  computer  interactive  option  to  query  the  film’s  characters  and  receive  scripted 
commentary.  Because  of  their  large  size,  the  digitized  video  files  are  played  from  the  hard  drive 
of  a  local  computer.  Each  vignette  file  takes  12-15  min  to  play  from  start  to  finish. 

The  family  of  AXL  materials  includes  theme-based  modules  and  an  instructor’s  guide 
(Zbylut  &  Ward,  2004b).  The  interactive  multimedia  modules  enable  small  groups  to  address 
various  teaching  themes  related  to  military  leadership.  The  instructor’s  guide  gives  instructions 
for  leading  vignette-based  training  sessions,  including  discussion  questions  and  tips  for  guiding 
the  discussion.  Among  the  modules  are  the  following; 

•  Command  climate 

•  Command  influence 

•  Communication 

•  Cultural  awareness 

•  Guiding  subordinates 

•  Mission  clarity 

•  Model  of  command 

•  Respect  for  experience 

•  Leadership  values 

During  the  operational  demonstration,  the  research  team’s  vignette  facilitator  adapted  the 
AXL  methodology  for  the  unit  training  environment,  yielding  the  following  steps: 

•  Introduction  to  the  training  session 

•  Viewing  the  AXL  vignette  film  (entire  group) 

•  Small  groups  (2-3  per  computer)  working  on  the  cultural  awareness  module 

•  Small  groups  (2-3  per  computer)  working  on  the  leadership  values  module 

•  Data  collection  (specialized  diagnostics  or  user  satisfaction) 

Training  Support  Tools 

The  research  team  developed  training  support  tools  to  guide  participants’  utilization  of 
the  toolkit  and  to  facilitate  proper  use  of  the  tools  for  optimal  learning.  These  tools — Student 
Guide,  Training  Support  Package,  and  Help  Desk  mechanism — are  summarized  below. 
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Student  Guide 


The  Student  Guide  served  to  eneourage  participation  in  the  ALD  program  and  facilitate 
proper  use  of  the  tools.  An  important  purpose  was  to  ensure  that  the  participants  received 
maximum  benefit  from  their  ALD  training.  As  seen  in  Appendix  C  (Volume  II),  the  guide 
included  an  introduction,  descriptions  of  all  the  tools,  instructions  for  the  participants/students, 
technical  notes,  and  information  for  contacting  the  research  team.  The  bulk  of  the  guide 
described  the  tools,  including  the  number  and  sequencing  of  training  modules  and  the  expected 
completion  time.  A  multimedia  version  of  the  guide  with  voice  track  was  available  on  the  ALD 
homepage,  and  the  participants  received  a  paper  copy  in  slide  format  during  the  unit’s  group 
orientation. 

Training  Support  Package 

The  Training  Support  Package  (TSP)  provided  a  detailed  guide  for  unit  administrators 
and  interested  participants.  It  included  an  overview  of  the  operational  demonstration  and  the 
assessment  process,  benefits  and  costs  of  using  the  toolkit,  the  training  model,  tool  descriptions, 
and  administrative  guidelines  for  the  demonstration.  Twelve  appendices  of  the  TSP  contained 
details  on  various  topics  important  to  the  demonstration,  to  include  Web  site  description  and 
guidelines  for  unit  training  schedules.  An  html  file  of  the  TSP  with  a  hyperlinked  index  resided 
on  the  Web  site,  and  a  text  file  was  available  for  exporting  and  printing.  Appendix  A  (Annex  G, 
page  A-33)  in  Volume  II  presents  the  text  version  of  the  TSP. 

The  administrative  guidelines  called  for  appointing  Unit  Action  Officers  (additional  duty) 
at  battalion  and  company  levels  to  facilitate  the  unit’s  utilization  and  assessment  of  the  program. 
These  positions  were  intended  to  be  used  only  during  the  operational  demonstration  and  would 
be  optional  for  routine  implementation. 

Help  Desk 

A  limited  Help  Desk  was  maintained  to  troubleshoot  technical  problems  and  answer 
questions  from  participants  and  unit  command  group  members.  A  “Help”  link  on  the  ALD  Web 
site  led  to  an  online  form  for  submitting  a  problem  report  or  help  request.  Participants  could  also 
submit  problem  reports  and  questions  via  email  and  telephone. 

Specific  Help  Desk  capabilities  included  correcting  malfunctions  in  the  Web  site  or  its 
links;  answering  Web  site  questions/issues;  assisting  with  training  support  issues;  and  providing 
guidance  on  the  ALD  program  materials.  As  necessary,  the  Help  Desk  administrator  relied  on  the 
Webmaster  and  tool-specific  experts  to  resolve  technical  issues.  A  record  of  Help  Desk  queries 
and  actions  was  maintained.  Additional  procedural  details  are  available  in  Appendix  B  (page  B- 
6)  of  Volume  11. 
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Delivery  Suite  (Web  Site) 


Overview 

The  components  of  the  ALD  toolkit  were  hosted  on  a  Web  site  server  and  managed  by  a 
learning  management  system  (LMS).  The  singular  exception  to  this  arrangement  was  the  AXL 
suite,  which  was  distributed  via  compact  disc  and  managed  by  a  member  of  the  research  team 
who  served  as  the  vignette  facilitator.  In  addition  to  enabling  internet  delivery  of  the  ALD  tools, 
the  Web  site  supported  the  following  functions:  account  set-up,  user  authentication,  learning 
management,  usage  monitoring,  problem  reporting,  assessment  data  collection,  data  storage  and 
management  (database),  and  administrative  functions. 

This  section  presents  a  high  level  description  of  the  Web  site.  For  more  details,  see 
Appendix  A  (Annex  I,  page  A-56)  in  Volume  II. 

Security  and  Privacy 

Controlling  access  to  the  Web  site  relied  on  a  user  authentication  mechanism  to  prevent 
unauthorized  access.  Each  participant  chose  a  unique  identifier  and  password  to  use  for  logins. 
For  data  security,  an  identification  number  was  assigned  to  each  participant  and  no  personal 
identifying  information  (e.g.,  social  security  number,  surname)  was  stored  in  the  database.  No 
record  linking  personal  identifying  information  with  data  was  maintained.  Access  to  the  contents 
of  the  database  was  strictly  limited  to  selected  members  of  the  research  team. 

Web  Site  Architecture 

The  Web  site  was  constructed  for  user  interface  presentation  via  commercially  available 
Web  browser  packages.  The  hypertext  pages  were  designed  for  easy  legibility,  clear  separation 
of  information  elements,  and  minimum  scrolling  requirements.  The  top-level  pages  for  entering 
the  system  and  accessing  the  ALD  tools  included: 

•  Homepage  with  user  login  portal,  “contact  us”  link,  and  Student  Guide  link 

•  Account  set-up  page  (presented  to  new  users  only) 

•  Security  and  privacy  notice  page 

•  Welcome  page  with  general  information  and  links  to  the  Student  Guide  and  the  TSP 

•  Main  menu  with  three  primary  sections: 

o  Pre  Measures — links  to  demographic  survey  plus  six  self-assessment  tools 
o  Training — LASTS  link  plus  individual  links  to  eight  CTS  modules 
o  Post  Measures — links  to  four  self-assessment  tools  plus  eight  CTS  post-tests 

The  ALD  homepage  could  be  accessed  directly  at  a  unique  address.  All  program 
components  resided  on  the  Web  site  server  except  the  Personality  Assessment  (accessed  on 
another  Web  site)  and  the  AXL  suite  (delivered  offline).  When  a  participant  selected  a  specific 
tool  on  the  main  menu,  the  link  typically  led  to  a  secondary  menu  of  modules  (or  components  of 
CTS  modules)  available.  Selecting  a  link  from  the  secondary  menu  led  to  a  family  of  tool- 
specific  pages  that  conveyed  the  contents  of  the  diagnostic  or  training  module. 
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The  research  team  selected  a  public  domain  LMS  (Moodle)  because  it  has  no  purchase  or 
licensing  costs  and  meets  the  Army’s  information  technology  standards  (including  compliance 
with  the  Sharable  Content  Object  Reference  Model).  It  has  been  used  successfully  on  other  DOD 
websites  and  was  recommended  by  military  training  experts.  The  LMS  also  managed  the  capture 
of  recordable  participant  responses  (including  diagnostic  data  and  training-related  measures)  and 
transferred  them  to  the  database. 

The  LMS  software  included  a  database  that  stored  and  managed  the  data  collected  during 
online  assessment  and  training.  Login  and  module  usage  parameters  (start/stop  clock  times)  also 
were  recorded  and  stored  in  the  database.  The  researchers  adapted  report  generation  functions  of 
the  LMS  software  for  monitoring,  data  analysis,  and  administrative  purposes. 

Exploiting  the  Web  Site 

Once  a  participant  established  an  account  on  the  ALD  Web  site,  he/she  could  login  and 
logout  at  will — 24  hr  a  day,  7  days  a  week.  There  was  no  minimum  or  maximum  limit  on  the 
duration  of  a  session,  nor  on  the  time  between  sessions.  Thus  the  participant  controlled  the 
timing  of  his/her  activities  in  working  with  the  ALD  tools.  The  participant  also  determined  his  or 
her  own  pace  during  a  session,  normally  by  selecting  “Next”  or  “Enter”  or  a  new  link  when 
he/she  was  ready  to  advance.  As  a  general  rule,  the  participant  could  work  through  part  of  a 
module  or  more  than  one  module  in  a  single  session.  However,  the  LMS  controlled  the  sequence 
of  ALD  tools. 

To  create  optimal  conditions  for  ALD  training  as  well  as  data  collection,  the  LMS 
enforced  the  following  sequence  of  the  diagnostic  and  training  tools:  Demographic  Survey,  UCI, 
Leadership  Experiences,  MLQ,  Eeadership  Opinions,  TKME,  Personality  Assessment,  EASTS, 
and  finally  CTS  modules.  After  finishing  the  EASTS  tool,  a  participant  could  access  the  eight 
CTS  modules  in  any  sequence.  Completing  a  given  tool  produced  a  check-mark  next  to  a  tool’s 
link  in  the  main  menu.  Participants  could  revisit  tools  completed  earlier,  if  desired. 

Due  to  the  design  features  of  the  ALD  tools,  book-marking  capabilities  were  generally 
absent.  As  a  result,  work  on  the  diagnostic  and  LASTS  tools  was  lost  if  the  participant  exited 
before  reaching  the  end  of  a  module.  A  warning  about  the  potential  loss  of  data  appeared  on  the 
opening  page  of  each  module  in  this  category.  In  addition,  a  few  tools  imposed  a  completion 
time  limit.  This  constraint  was  removed  midway  through  the  operational  demonstration. 

When  a  tool  yielded  feedback  for  the  participant,  it  was  normally  delivered  online.  An 
exception  to  this  practice  was  the  Personality  Assessment,  where  the  narrative  report  was  sent  to 
an  email  address  designated  by  the  participant. 

Utilization  Model 


Training  Model 

To  guide  the  utilization  of  the  ALD  tools  in  a  consistent  fashion  likely  to  produce  optimal 
program  benefits,  the  research  team  crafted  a  training  model.  The  training  model  addressed  two 
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dimensions — sequence  and  time.  The  start  of  the  sequence  rested  on  the  argument  that  self- 
awareness  and  strategic  thinking  about  self-development  are  basic  enablers  (Day,  Harrison,  & 
Halpin,  2009).  This  premise  suggested  that  the  self-assessment  tools  and  the  LASTS  modules 
should  come  first  to  give  the  individual  an  appreciation  of  his/her  strengths  and  leader 
development  approach.  With  an  objective  foundation  identified,  a  mixture  of  individual  training 
on  cognitive  capabilities  (CTS)  and  group  training  on  interpersonal  skills  (AXL)  emerged  as  a 
blended  learning  application.  Staging  the  two  types  of  training  to  build  progressively  across  the 
individual  and  group  domains  could  leverage  socially  situated  learning  and  consolidation  of 
knowledge  and  skills.  Finalizing  the  sequence  logically  entailed  post-training  diagnostics  to 
probe  for  evidence  of  learning  effects.  This  train  of  logic  led  to  the  recommended  sequence 
depicted  in  Figure  3. 


Timeline 

Start  ^  ^  ^  ^  ^  Finish 

Activity 

Diagnostics 

Training 

Diagnostics 

Tools 

Pre- 

Measures 

LASTS 

CTS  1-4 

AXL#1 

CTS  5-8 

AXL  #2 

Post- 

Measures 

Time 

4-6  hr 

3-4  hr 

8-10  hr 

2-3  hr 

8-10  hr 

2-3  hr 

3-4  hr 

Figure  3.  Training  sequence  as  recommended  for  the  operational  demonstration. 

The  time  dimension  of  the  training  model  was  intended  to  shape  realistic  expectations  for 
the  level  of  effort  needed  to  fully  harness  the  ALD  toolkit.  The  research  team  drew  on  the 
formative  evaluation  results  from  the  development  efforts  for  the  various  tools,  and  on  feedback 
gathered  during  the  pilot  test.  This  led  to  the  time  parameters  shown  in  Figure  3,  reinforced  by 
including  time  estimates  for  each  tool  in  the  Web  site’s  main  menu.  Implementing  the  time  and 
sequence  features  of  the  training  model  could  be  expected  to  optimize  the  benefits  from  working 
with  the  components  of  the  ALD  toolkit. 

The  model  represented  in  Figure  3  provided  working  guidelines  for  units  and  participants 
interested  in  putting  the  ALD  toolkit  to  good  use.  Although  the  LMS  enforced  a  fixed  sequence 
for  the  diagnostic  and  LASTS  tools,  the  system  allowed  participants  to  use  the  subsequent  tools 
in  any  sequence  desired.  Also,  participants  could  return  to  a  module  for  review  or  further  study, 
and  they  were  encouraged  to  do  so. 

Optimally,  ALD  training  would  occur  primarily  during  the  reset  phase  of  the  unit 
lifecycle.  However,  the  training  modules  are  suitable  for  use  during  other  phases  of  the  lifecycle 
including  individual  and  team  training,  certifying  events  at  Combat  Training  Centers,  and 
sustainment  training.  As  the  recruitment  of  units  for  the  operational  demonstration  worked  out, 
there  was  little  opportunity  to  align  unit  participation  with  a  particular  phase  of  the  lifecycle. 
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Utilization  Environment 


The  ALD  tools  were  intended  for  use  by  eompany  grade  leaders  during  duty  hours  as  part 
of  the  unit’s  training  program.  The  ALD  arehitects  envisioned  that  users  would  have  ready 
aceess  to  eomputers  within  the  unit  or  in  a  nearby  learning  eenter.  In  reality,  the  participating 
units  had  few  computers  and  most  of  them  were  not  readily  accessible  to  participants. 

The  operational  demonstration  plan  envisioned  individual  work  with  the  toolkit  to  be 
included  on  the  unit  training  schedule,  and  the  TSP  contained  materials  to  facilitate  such 
scheduling.  However,  only  the  AXL  events  were  placed  on  the  training  schedule.  Consequently, 
participants  used  the  other  tools  as  their  schedules  permitted.  Consistent  with  the  COE,  the  hectic 
pace  of  unit  activities  (individual  and  collective  training  events,  equipment  management  and 
maintenance,  administrative  functions,  external  taskers,  and  so  on)  severely  limited  the  time 
participants  had  available  to  spend  on  ALD  training. 

Demonstration  and  Assessment  Methods 

This  section  describes  the  documentation,  research  staffing,  participants,  investigational 
paradigms,  procedures,  and  data  collection  processes  employed  in  planning  and  executing  the 
ALD  demonstration  and  assessment.  The  questions  of  interest,  sampling  approach,  assessment 
design,  evaluation  approach,  and  measurement  strategy/approaches  are  described  in  the  Detailed 
Test  Plan  (see  Appendix  A  in  Volume  II).  The  description  here  focuses  on  the  actual  methods 
executed,  regardless  of  those  called  for  in  the  Detailed  Test  Plan  and  other  planning  documents. 
Notable  departures  from  the  planned  methods  occurred  on  a  number  of  counts  to  accommodate 
the  requirements  and  constraints  of  the  operational  environment. 

Overview 

The  overarching  approach  called  for  conducting  an  operational  demonstration  under 
realistic  field  conditions  and  gathering  feedback  from  members  of  the  target  audience.  To  ensure 
an  independent  assessment,  none  of  the  ALD  tool  developers  participated  directly  in  planning 
and  executing  the  demonstration  and  data  collection.  The  research  team  created  a  Web  site  to 
enable  online  delivery  of  the  ALD  tools  and  pilot  tested  the  site  before  opening  it  to  outside 
access.  The  team  targeted  operational  units  in  both  the  Active  Component  (AC)  and  Reserve 
Component  (RC),  but  only  AC  units  participated.  The  investigators  also  worked  with  user  juries 
to  expand  the  feedback  base.  The  operational  users  and  user  juries  yielded  data  on  the  suitability 
of  the  tools,  utilization  patterns,  basic  learning,  module-specific  performance,  user  acceptability, 
and  improvement  options.  Data  analysis  served  to  characterize  the  utilization  of  the  ALD  tools 
and  to  derive  lessons  learned  that  could  help  improve  the  interactive  self-development  program. 

The  following  steps  were  executed  to  accomplish  the  operational  demonstration  and 
assessment  objectives: 

•  Thorough  demonstration  and  assessment  planning 

•  Development  of  an  integrated  Web  site  (see  Woller  et  ah,  in  preparation) 

•  Pilot  testing  of  the  Web  site,  with  subsequent  refinement 

•  Planning,  preparing,  executing,  and  documenting  user  juries 
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♦  Planning,  preparing,  executing,  and  documenting  the  operational  demonstration 

♦  Compiling,  integrating,  and  analyzing  the  cumulative  data 

Planning  Documents 

Through  a  systematic  planning  process  the  research  team  designed  the  investigational 
approach,  developed  the  implementation  procedures,  identified  measurement  objectives  and 
methods,  and  determined  resource  requirements.  This  largely  analytical  process  produced  three 
planning  documents  to  guide  the  operational  demonstration  and  assessment: 

♦  Detailed  Test  Plan 

♦  Implementation  Support  Plan 

♦  Data  Collection,  Management  and  Analysis  Plan 

The  Detailed  Test  Plan  (see  Appendix  A  in  Volume  II  of  this  report)  established  the 
framework  and  architecture  for  the  operational  demonstration  and  set  the  stage  for  creating  the 
subsequent  planning  documents.  Development  of  the  Detailed  Test  Plan  began  by  identifying  the 
purpose,  goals  and  objectives  of  the  demonstration  and  assessment.  The  objectives  were 
translated  into  key  questions  of  interest.  Based  on  the  goals  and  questions  of  interest,  the 
research  team  developed  an  integrated  assessment  approach,  including  a  sampling  approach  and 
measurement  strategy.  The  investigators  derived  the  measurement  requirements  and  mapped 
them  to  a  correlated  family  of  measurement  approaches,  identifying  the  potential  sources  of  data. 
In  addition  to  the  research  plan,  the  Detailed  Test  Plan  included  a  management  plan  addressing 
timeline  aspects,  management  structure,  recruiting  of  units,  demonstration  strategy,  resource 
requirements,  decision  milestones,  and  finally  threats  to  the  validity  of  the  assessment  along  with 
approaches  for  avoiding  the  threats  or  mitigating  their  impact. 

The  Implementation  Support  Plan  (see  Appendix  B  in  Volume  II)  spelled  out  in  detail  the 
blueprint  for  meeting  the  objectives  of  the  operational  demonstration.  The  document  outlined  the 
stages  and  steps  that  formed  the  heart  of  the  blueprint.  After  defining  the  target  audience,  it  then 
detailed  the  key  operational  procedures:  overall  management,  Web  site  management,  help  desk 
operations,  unit  participation,  monitoring  procedures,  and  information  management.  Next  came  a 
timeline  for  the  stages  and  steps  of  the  operational  demonstration.  The  document  carefully 
defined  the  personnel  requirements  and  their  associated  duties  and  qualifications.  A  risk 
mitigation  plan  outlined  measures  to  control  or  offset  the  factors  that  could  threaten  the  success 
of  the  demonstration.  Finally,  the  document  included  an  outline  plan  for  the  pilot  test. 

The  Data  Collection,  Management  and  Analysis  Plan  set  the  terms  for  capturing  and 
processing  both  quantitative  and  qualitative  metrics.  The  goal  was  to  obtain  data  capable  of 
answering  the  research  questions  of  interest.  Using  the  key  questions  from  the  Detailed  Test  Plan 
(page  A-1 1  of  Volume  II),  the  research  team  inventoried  the  data  requirements  and  translated 
them  into  a  master  list  of  measures  and  other  metrics.  They  then  mapped  the  master  list  of  data 
requirements  into  a  list  of  candidate  data  collection  instruments.  The  investigators  outlined 
operational  processes  for  collecting  data  and  organized  them  along  an  ideal  timeline  for  a 
participating  unit.  They  spelled  out  the  procedures  for  managing  the  data  and  organizing  them  in 
a  cumulative  database.  Finally,  they  outlined  the  approaches  for  analyzing  the  data.  As  the 
operational  demonstration  and  assessment  unfolded,  the  research  team  adjusted  the  measurement 
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procedures  to  fit  the  actual  utilization  conditions.  This  involved  ehoosing  the  relevant  and 
measurable  data  elements,  then  developing  the  supporting  data  oolleetion  instruments. 


Pilot  Test 

To  test  the  funetionality  of  the  ALD  Web  site  and  ensure  readiness  for  the  operational 
demonstration,  the  researeh  team  eondueted  a  pilot  test  of  all  primary  and  supporting  materials. 
The  objectives  were  to  verily  that  the  ALD  produets,  the  Web  site  and  its  supporting  software, 
researeh  team  personnel,  training  support  materials  and  proeedures,  and  data  eolleetion  and 
management  meehanisms  were  ready  for  full  exeeution.  The  test  exereised  key  proeedures 
speeified  in  the  Implementation  Support  Plan.  During  this  event  independent  researehers  tested 
the  structure  and  functions  of  the  Web  site  as  well  as  the  proeess  for  gathering  utilization  data 
and  user  feedbaek.  Table  4  summarizes  the  prineipal  steps  of  the  pilot  test  methodology. 

Table  4 

Primary  Steps  of  the  Pilot  Test 


Phase 

Step 

Preparation 

•  Schedule  participants  to  represent  the  target  audience 

•  Develop  pilot  test-specific  materials  and  instruments 

•  Confirm  that  Web  site,  hardcopy  materials,  and  participants  are  ready 

Execution 

•  Conduct  kick-off  briefing  as  the  pilot  test  launch  event 

•  Sequentially  test  components  and  document  problems/observations 

•  Hold  as-needed  hotwashes  to  surface  problems  of  immediate  concern 

Documentation 

•  Gather  data  collection  instruments  from  all  participants 

•  Compile  problems,  findings,  lessons  learned,  and  recommendations 

•  Inventory  and  prioritize  follow-on  actions  required 

The  pilot  test  entailed  a  limited  trial  with  all  players  (espeeially  sourees  of  data) 
represented — demonstration/assessment  manager,  operations  manager,  Web  site  experts,  users 
(target  audienee),  data  eolleetors,  and  help  desk  personnel.  The  role  players  were  eontraet 
personnel  loeated  at  four  sites — Forts  Leavenworth,  Knox,  Benning,  and  Hood.  A  ten-day 
sehedule  eovered  Web  site  loading  (stress  test),  employment  of  pre-training  diagnostics, 
utilization  of  individual  training  tools,  conduct  of  collective  exereises  (AXL),  and  deployment  of 
post-training  diagnosties.  The  sehedule  also  ineluded  the  use  of  the  primary  user  feedbaek  tool 
(user  satisfaetion  survey)  and  pilot  test-specifie  data  eolleetion  instruments  (worksheet  and 
partieipant  journal).  The  test  eoneluded  with  an  after  aetion  review  to  summarize  the  major 
lessons  learned  as  well  as  follow-on  aetions. 

The  entire  proeess  for  oolleeting  and  managing  data  (ineluding  database  operations)  was 
exereised,  and  data  analysis  proeedures  were  tested  seleetively.  Investigators  eheeked  the  quality 
of  the  data  eapture  proeesses,  espeeially  the  automated  meehanisms.  During  testing  of  the  ALD 
tools,  usage  and  performanee  data  populated  into  the  database  by  means  of  automated  Web  site 
proeesses  were  inspeeted  for  tagging,  aecuraey,  and  eompleteness  of  data  elements.  Data  entered 
manually  were  spot  eheeked  randomly.  Flaws  and  problems  were  fixed  and  retested. 
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The  investigators  used  the  results  of  the  pilot  test  to  revise  the  Web  site — including  the 
user  interface,  selected  diagnostic  tools,  the  learning  management  process,  and  the  data  capture 
mechanisms.  They  also  revised  the  training  support  materials,  the  data  collection  instruments, 
and  the  data  management  procedures.  In  addition,  they  refined  the  procedures  for  managing  user 
access,  monitoring  Web  site  use,  and  resolving  problems. 

User  Juries 

To  provide  narrowly  focused  feedback  opportunities,  the  research  team  planned  and 
conducted  a  series  of  user  juries.  A  user  jury  is  a  group  of  target  audience  representatives  that 
previews  or  reviews  a  new  product  or  system  in  a  compressed  timeframe  and  provides  limited 
feedback  (Throne,  2006).  In  the  ALD  assessment,  the  user  jury  technique  served  to  expose  a 
captive  audience  to  a  selected  component  of  the  toolkit  in  a  well  controlled  setting.  While  the 
technique  did  not  represent  a  fully  operational  test,  it  enabled  the  research  team  to  investigate 
specific  dimensions  of  ALD  tools  such  as  technical  quality,  user  acceptability,  doctrinal 
suitability,  and  potential  value. 

The  investigators  harnessed  the  user  jury  sessions  to  assess  two  of  the  three  ALD  training 
tools — LASTS  and  CTS  modules.  (Captive  audience  implementations  of  the  AXL  vignettes  were 
built  into  the  unit  assessment  process.)  Across  14  user  juries  at  five  different  installations  (Table 
5),  the  research  team  assessed  most  of  the  LASTS  modules  and  all  of  the  CTS  modules.  With  the 
exception  of  two  of  the  CTS  modules,  a  given  module  was  assessed  only  once. 

Table  5 

User  Jury  Sessions  Conducted  during  the  Demonstration 


User  Juries 

Tool  Examined 

Data  Collection 

Location  1  (4  groups) 

LASTS  Modules  (4  of  5) 

Demography,  Think-Aloud,  Surveys,  Hotwash 

Location  2  (2  groups) 

CTS  Modules  #1  and  #2 

Demography,  Pre-  &  Post-Tests,  Surveys 

Location  3  (2  groups) 

CTS  Modules  #3  and  #4 

Demography,  Pre-  &  Post-Tests,  Surveys,  Hotwash 

Location  4  (2  groups) 

CTS  Modules  #5  and  #6 

Demography,  Pre-  &  Post-Tests,  Surveys 

Location  5  (4  groups) 

CTS  Modules  #7  and  #8 

Demography,  Pre-  &  Post-Tests,  Surveys 

The  research  team  adopted  a  familiarize-inspect-refiect  process  for  conducting  the  user 
jury  sessions.  Each  group  of  troops  addressed  only  one  module,  so  that  no  jury  members  worked 
with  more  than  one  ALD  product.  Each  session  lasted  3-5  hr  and  included  an  introduction  to  the 
AED  program,  an  overview  of  the  module  of  interest  (familiarization),  observation  of  the  module 
in  group  mode  (inspection),  and  subsequent  assessment  and  discussion  (reflection).  During  the 
inspection  phase,  the  group  of  participants  (from  6  to  15  Captains,  Eieutenants  or  NCOs  on 
active  duty)  walked  through  the  module  of  interest  by  viewing  the  user  interface  on  a  large 
projection  screen.  As  the  research  facilitator  led  the  group  through  the  multimedia  screens,  the 
participants  verbalized  their  reactions  and  opinions.  After  the  walk-through,  the  jury  members 
provided  reflection-based  feedback  by  completing  a  questionnaire  and  participating  in  a  group 
discussion  or  hotwash.  As  part  of  the  survey  process,  the  participants  furnished  demographic 
information  that  included  rank,  branch,  current  assignment,  and  years  in  the  Army. 
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The  user  satisfaetion  survey  (see  Appendix  D  in  Volume  II  for  versions  tailored  to 
LASTS,  CTS,  and  unit  partieipants)  eontained  self-report,  rating  seale,  and  write-in  items  to 
elicit  reactions  and  opinions  from  the  participants.  The  opinion  rating  items  used  a  6-point  or  7- 
point  agree-disagree  response  scale.  The  questionnaires  addressed  the  following  dimensions  of 
the  ALD  tools: 

•  Overall  impressions  (potential  role,  warranted  investment  of  time,  relative  priority) 

•  Technical  quality  (accuracy,  clarity,  conciseness,  quality  of  feedback,  etc.) 

•  Acceptability  (ease  of  use,  sensibility,  realism,  level  of  challenge,  etc.) 

•  Learning  contributions  (e.g.,  awareness,  knowledge,  insights,  understanding,  skills) 

•  Potential  value  (relevance,  importance,  applicability,  usefulness,  benefits,  etc.) 

•  Cost-benefit  aspects  (e.g.,  global  return  on  investment  of  time) 

•  Potential  leader  development  issues  the  ALD  tools  might  address 

•  Ideas  and  suggestions  for  improving  the  modules  or  the  overall  program 

The  facilitator  recorded  hand-written  notes  of  the  think-aloud  verbalizations  and  group 
discussions,  later  compiling  them  into  a  thematic  summary.  Occasionally  another  researcher 
served  as  a  note-taker.  The  participants  completed  the  questionnaires  in  paper-and-pencil  form. 

In  the  case  of  the  CTS  modules,  the  pre-  and  post-tests  from  the  toolkit  were  administered  to  the 
participants  in  hardcopy  form. 

A  total  of  140  Soldiers  participated  in  the  user  juries.  This  included  48  Captains,  23  First 
Lieutenants,  52  Second  Lieutenants,  5  Sergeants  First  Class,  5  Staff  Sergeants,  and  7  Sergeants. 
The  participants  represented  various  branches  including  Armor/Cavalry,  Infantry,  Field  Artillery, 
Military  Intelligence,  Air  Defense  Artillery,  Aviation,  Ordnance,  Military  Police,  Engineer, 
Quartermaster,  Transportation,  Signal,  Chemical,  Adjutant  General,  Medical  Service,  and  Judge 
Advocate  Corps.  The  time  on  active  duty  ranged  from  3  months  to  20  years  (Mdn  =  6.4  years). 

Operational  Demonstration 

The  Implementation  Support  Plan  guided  the  field  testing  (see  Appendix  B  in  Volume  II). 
The  plan  spelled  out  the  methods  and  procedures  for  executing  the  operational  demonstration.  At 
a  strategic  level  the  plan  called  for  enlisting  tactical  units  to  participate  in  the  demonstration, 
giving  them  guidelines  for  utilizing  the  various  tools  on  the  Web  site,  measuring  indicators  of 
utilization  and  impact,  and  gathering  feedback  from  the  users.  The  plan  organized  the 
demonstration  and  assessment  activities  into  the  stages  and  steps  outlined  in  Table  6. 

The  primary  research  personnel  who  supported  the  operational  demonstration  and 
assessment  included: 

•  Demonstration/assessment  manager  (primary  interface  with  participating  units) 

•  Operations  manager  to  coordinate  day-to-day  activities 

•  Web  site  cell  (designer/manager,  developer/programmer,  database  specialist) 

•  AXL  facilitator  (the  demonstration/assessment  manager)  to  lead  vignette  sessions 

•  Help  desk  administrator  to  resolve  operational  and  technical  problems 

•  Data  collector  to  conduct  interview  and  survey  sessions  on-site 
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Two  battalions  participated  in  the  operational  demonstration  and  assessment.  Both  were 
Aetive  Component  combat  units  based  at  home  stations.  Within  these  units  the  target  audience 
for  using  the  ALD  toolkit  included  “leader  teams”  within  the  company  formation  (platoon 
leaders,  eompany  commanders  and  executive  offieers  [XOs],  and  non-commissioned  officers 
[NCOs]  in  the  grade  of  Staff  Sergeant  and  above).  The  eligible  population  totaled  approximately 
80  personnel  in  eaeh  battalion.  For  the  two  units  the  demonstration  period  was  7  months  and  4 
months,  respeetively,  to  aecommodate  the  heavy  OPTEMPO.  Seheduling  was  coordinated  with 
the  battalion  training  oflioer  (S3). 

Table  6 

Primary  Steps  of  the  Operational  Demonstration 


Stage 

Step 

Preparation 

•  Recruit  brigade  combat  teams  or  battalions  to  participate 

•  Update  training  support  materials  (Student  Guide,  TSP) 

•  Develop  unit  orientation/start-up  package  (briefing  slides) 

•  Establish  help  desk  capabilities  (email  and  telephone) 

•  Operationalize  ALD  Web  site  (with  improvements  based  on  pilot  test) 

Unit  Start-up 

•  In-brief  commander  and/or  other  members  of  command  group 

•  Orient  participating  leaders  and  initialize  utilization  process 

•  Verify  unit  access  to  ALD  Web  site  and  help  desk 

•  Resolve  concerns  and  issues  from  users  and  leaders 

Toolkit  Usage 

•  Maintain  Web  site  operations  to  enable  participants’  use  of  tools 

•  Monitor  participants’  online  usage  and  encourage  unit  leaders 

•  Provide  self-assessment  feedback  to  individual  participants 

•  Resolve  technical  and  operational  problems 

•  Conduct  AXL  training  sessions  (with  ARI  facilitator  on-site) 

•  Collect,  process,  and  manage  online  and  paper-based  data 

Wrap-up 

•  Assess  status  of  data  collection  and  determine  logical  stop  point 

•  Gather  end-of-demonstration  feedback  from  users  and  unit  leaders 

•  Provide  overall  summary  to  users  and  unit  leaders 

Between  the  two  battalions  that  supported  the  demonstration  and  assessment,  about  160 
leaders  were  eligible  to  partieipate  in  ALD  training  and  data  collection.  Of  those,  142  leaders 
partieipated  in  one  or  more  ALD  training  activities.  Of  the  total  group,  76  took  part  in  only  the 
offline  AXL  events,  meaning  they  left  no  traee  on  the  Web  site.  Thus,  66  leaders  spent  time  on 
the  ALD  Web  site,  with  53  of  them  contributing  demographic  data.  All  were  in  active  duty 
status,  and  all  but  one  were  males.  The  group  included  2  Lieutenant  Colonels,  2  Majors,  8 
Captains,  6  Lirst  Lieutenants,  16  Second  Lieutenants,  4  Master  Sergeants,  9  Sergeants  Lirst 
Class,  2  Staff  Sergeants,  and  4  Sergeants.  Their  time  on  active  duty  ranged  from  10  months  to  23 
years  (Mdn  =  8.5  years).  Altogether  the  leaders  represented  eight  branehes — Infantry,  Armor, 
Lield  Artillery,  Military  Intelligence,  Aviation,  Signal,  Military  Police,  and  Adjutant  General. 

In  the  unit  start-up  stage,  the  demonstration/assessment  manager  initially  eondueted  a 
deskside  in-brief  for  the  commander  and  other  members  of  the  command  group  to  introduce  the 
ALD  program  and  the  goals  of  the  operational  demonstration.  The  package  for  the  in-brief  drew 
on  the  materials  in  the  TSP  and  Student  Guide  and  highlighted  the  training  time  needed  by 
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participants  to  work  with  the  online  ALD  products.  The  demonstration/assessment  manager  then 
eondueted  a  group  orientation  for  the  target  audienee  explaining  the  purpose  of  training,  aeeount 
set-up,  and  the  reeommended  sequenee  for  using  the  tools.  The  orientation  used  the  Student 
Guide  (in  slide  form)  to  give  participants  an  appreciation  of  the  ALD  toolkit,  as  well  as  narrative 
deseribing  the  purpose  of  the  demonstration  and  the  projected  timeline.  A  primary  objeetive  of 
the  in-brief  and  orientation  was  to  generate  support  for  the  operational  demonstration. 

As  a  general  rule  eaeh  partieipating  unit  had  a  single  point  of  eontaet  to  serve  as  the 
primary  interfaee  between  the  unit  and  the  research  team.  This  individual  eneouraged  the  unit’s 
eligible  leaders  to  use  the  ALD  toolkit,  and  helped  resolve  issues  that  eould  affeet  partieipation. 
The  demonstration/assessment  manager  maintained  routine  eontaet  with  the  unit  points  of 
eontaet,  keeping  them  informed  of  the  partieipation  status  within  their  units. 

Eaeh  unit  partieipant  initially  ereated  an  aeeount  on  the  ALD  Web  site,  ehoosing  a  user 
name  and  password.  The  user  interfaee  invited  partieipants  to  review  the  Student  Guide  (and  the 
TSP,  if  more  information  was  desired)  before  starting  any  modules.  Subsequently  they  could 
work  on  modules  at  their  own  paee,  following  the  sequenee  enforeed  by  the  LMS.  As  a  general 
rule,  the  LMS  required  users  to  eomplete  the  pre-training  diagnostic  modules  first,  then  the 
LASTS  modules,  then  the  CTS  modules  (in  any  order),  and  finally  the  post-training  diagnostie 
modules.  The  users  eould  revisit  the  Student  Guide  or  the  TSP  as  desired.  The  online  suite’s 
strueture  and  the  LMS  proeess  are  deseribed  in  Appendix  A  (Annex  I,  page  A-56)  in  Volume  II, 
and  teehnieal  details  ean  be  found  in  Woller  et  al.  (in  preparation). 

As  the  demonstration  and  assessment  proeeeded,  user  feedbaek  occasionally  led  to  minor 
improvements  in  seleeted  features  of  the  online  suite.  These  improvements  generally  revolved 
around  wording  or  highlighting  ehanges  in  the  lead-in  instruetions,  the  main  menu  links,  the 
Student  Guide,  the  TSP,  or  seleeted  diagnostic  tools.  In  addition,  the  help  desk  features  were 
refined  and  expanded.  Toward  the  end  of  the  primary  data  eollection,  the  LMS  eontrols  were 
modified  to  allow  random  aeeess  to  the  online  modules  (in  antieipation  of  an  additional 
demonstration  phase).  The  in-proeess,  evolutionary  improvements  to  the  online  suite  are 
doeumented  in  a  eompanion  report  (Woller  et  ah,  in  preparation). 

The  help  desk  proeess  relied  primarily  on  users  submitting  teehnieal  and  operational 
problems  via  email.  A  Web  site  link  eonneeted  users  to  a  pre-defined  email  form.  The 
demonstration/assessment  manager  normally  proeessed  an  incoming  email  to  determine  if  she 
eould  resolve  the  problem  direetly  or  needed  special  assistance.  Users  also  eould  submit  a 
regular  email  or  leave  a  telephone  message.  Onee  a  resolution  or  definitive  reply  had  been  sent  to 
the  originator,  the  administrator  eould  transfer  key  information  to  a  lessons  learned  folder. 

Participant  adherence  to  the  ALD  training  model  obviously  impaeted  the  validity  of  the 
data.  The  demonstration/assessment  manager  sent  email  reminders  to  the  partieipants  at  several 
points — ^primarily  with  eaeh  pre-measure  feedbaek  paeket  and  preeeding  eaeh  AXL  session.  In 
the  final  analysis,  it  proved  very  difficult  to  influence  partieipants’  behaviors  and  the  amount  and 
quality  of  the  resulting  data  reflected  substantial  variability  in  the  level  of  eomplianee. 
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Unlike  the  individual  learning  sessions  with  the  online  tools,  the  AXL  vignette  sessions 
were  eondueted  offline  in  group  fashion,  as  designed.  The  on-site  faeilitator  (demonstration/ 
assessment  manager)  led  a  group  of  partieipants  numbering  8-12  through  viewing  of  the  12-15 
min  vignette  on  a  laptop-driven  projection  screen.  The  participants  then  divided  into  groups  of  2- 
3  seated  around  a  computer.  Working  together,  each  group  completed  two  computer-based 
training  modules — one  on  cultural  awareness  and  one  on  leadership  values.  Each  session  lasted 
2-3  hr,  with  6  weeks  and  16  weeks,  respectively,  between  the  first  and  second  sessions  for  the 
two  different  battalions.  At  the  end  of  the  first  session,  the  facilitator  administered  two  tests 
measuring  participant  characteristics — Need  for  Cognitive  Closure,  and  Learning  Strategies.  The 
first  of  these  was  developed  as  part  of  the  AXL  project  (Zbylut  &  Ward,  2004a).  The  Learning 
Strategies  instrument  was  adapted  from  Duncan  and  McKeachie  (2005).  The  second  vignette 
session  ended  with  a  feedback  questionnaire  addressing  the  entire  ALD  toolkit. 

The  Data  Collection,  Management  and  Analysis  Plan  guided  the  capture  of  data  in  the 
operational  assessment.  As  Table  7  shows,  the  research  team  used  a  variety  of  data  collection 
instruments  that  ranged  from  manual  forms  to  automated  mechanisms.  Live  kinds  of  data 
collection  instruments  were  used — event  chronologies,  product-embedded  tests,  forced-choice 
diagnostics,  attitude  surveys  (questionnaires),  and  structured  interviews.  The  product-embedded 
tests  and  the  diagnostic  tools  were  utilized  as  is  (without  modifying  the  developer’s  final 
version).  The  user  satisfaction  surveys  (see  Appendix  D  in  Volume  II)  and  the  demographic 
survey  were  developed  by  the  investigators  through  iterative  review  and  revision  cycles. 

Table  7 

Data  Collection  Instruments  Used  during  the  Operational  Demonstration 


Instrument 

Description 

Timeframe 

Participant  Roster 

Listing  of  participants  (eligible  vs.  registered) 

Unit  start-up 

Web  Site  Logger 

Automated  documentation  of  login  events 

Continuous 

Product-Embedded  Tests 

Assessment  components  built  into  ALD  tools 

LMS-triggered 

Demographic  Survey 

Biographical  profile  completed  via  Web  site 

User  registration 

Toolkit  Diagnostics 

Pre-  and  post-training  assessment  via  Web  site 

Pre-  and  post-training 

User  Satisfaction  Surveys 

Gathering  of  participant  feedback  on-site 

End  of  training 

Interview  Protocol 

Conducting  interviews  on-site  (face-to-face) 

Intermittent 

Help  Desk  Log 

Recording  of  problems  and  their  resolutions 

Event-triggered 

In  the  absence  of  control  (baseline)  units,  the  research  team  used  a  pre-test  and  post-test 
paradigm  to  probe  for  effects  of  utilizing  the  ALD  training  tools.  Among  the  diagnostic  tools 
described  earlier  in  The  Leader  Development  Toolkit  section,  four  instruments  were  selected  for 
pre  versus  post  comparison  because  of  their  expected  sensitivity  to  self-development  learning: 

•  Multifactor  Leadership  Questionnaire 

•  Tacit  Knowledge  for  Military  Leaders 

•  Leadership  Opinions 

•  Unit  Cohesion  Index 
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Among  the  toolkit’s  diagnostic  tools,  two  were  deemed  unsuitable  for  pre-post  testing 
due  to  the  inherent  stability  of  the  measures  involved.  As  a  factual  account  of  one’s  leadership 
history  in  the  Army,  the  Leadership  Experiences  survey  could  not  be  expected  to  reveal  short¬ 
term  learning  effects.  As  an  instrument  designed  to  measure  stable  personality  dimensions,  the 
Personality  Inventory  would  not  be  a  candidate  to  reflect  the  acquisition  of  new  leadership 
knowledge,  skills,  and  understanding.  Thus,  these  two  diagnostic  instruments  served  mainly  as 
tools  to  enhance  a  leader’s  self-awareness  and  generate  baseline  measures. 

As  an  adjunct  to  the  set  of  post-training  diagnostic  tools,  the  research  team  added  the 
post-tests  from  the  CTS  modules.  They  did  this  for  two  reasons.  First,  there  was  interest  in 
examining  the  potential  value  of  the  tests  as  independent  diagnostic  tools.  Second,  the  team 
hypothesized  that  positive  effects  of  ALD  training  subsequent  to  completing  the  CTS  modules 
might  be  detectable  by  repeating  the  CTS  post-tests.  In  effect,  this  represented  an  extension  of 
the  pre-post  comparison  paradigm,  with  the  CTS  post-tests  to  be  used  during  training  and  again 
at  the  end. 

As  a  unit’s  participation  window  ended,  the  demonstration/assessment  manager 
interviewed  leaders  to  gather  insights  about  the  overall  ALD  program  and  the  issues  faced  in 
utilizing  the  toolkit.  She  also  provided  them  global  feedback  on  the  outcomes  of  the  assessment, 
encouraging  them  to  share  the  information  with  their  participating  leaders. 

Data  Handling  and  Analysis 

The  cumulative  data  included  qualitative  and  quantitative  measures  originating  from  the 
user  juries  and  the  operational  users.  Because  the  user  jury  data  were  not  comparable  in  terms  of 
procedures  and  measures,  they  were  not  compiled  into  a  common  database.  In  processing  the 
operational  users’  data,  the  research  team  aggregated  them  into  a  multi-dimensional  database. 
The  data  collected  via  online  processes  were  transferred  automatically  into  the  database.  The 
paper-and-pencil  and  hardcopy  data  were  entered  manually  and  spot  checked  for  accuracy. 

Parallel  procedures  were  used  for  analyzing  and  integrating  the  data  from  the  user  juries 
and  the  unit  participants.  Given  the  mixed  nature  of  the  data,  both  quantitative  and  qualitative 
techniques  were  used  for  analysis.  The  quantitative  analytical  techniques  were  limited  to 
descriptive  statistics.  The  members  of  the  research  team  liberally  applied  their  own  observations 
and  judgment,  especially  to  interpret  the  data  and  inject  investigators’  insights. 


Findings  and  Discussion 

The  presentation  of  findings  considers  data  from  unit  participants  and  user  juries  together 
to  shed  light  on  six  different  aspects  of  the  ALD  methodology  and  the  training  outcomes: 

•  The  program  of  instruction 

•  The  learning  environment 

•  Online  usage  of  ALD  tools 

•  Suitability  of  the  tools 

•  Learning  effectiveness  of  the  tools 
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•  Impact  and  value  of  using  the  tools 

Before  beginning  the  presentation  of  findings,  some  interpretive  eautions  are  in  order.  In 
the  complex  operational  environment  of  the  ALD  demonstration,  diverse  factors  could  influence 
the  quality  and  completeness  of  the  data  (Table  8).  Unwanted  faetors  could  compromise  the 
assessment’s  validity  (Cook  &  Campbell,  1979).  Most  or  all  of  the  unwanted  faetors  in  Table  8 
oceurred  during  the  operational  demonstration,  and  they  could  confound  the  analysis  and 
interpretation  of  the  resulting  data.  The  reader  should  be  aware  of  these  factors  in  considering  the 
findings  of  the  assessment. 

Table  8 

Unwanted  Factors  and  Their  Potential  Impact 


Unwanted  Factor 

Potential  Impact 

Unit-Related  Factors 

Shortfall  in  enrollment  of  units 

Low  confidence  in  findings,  inability  to  generalize  findings 

Sampling  of  active  duty  units  only 

Inability  to  draw  conclusions  about  Reserve  Component  units 

Absence  of  control/baseline  units 

Difficulties  discerning  true  effects  of  ALD  training 

Low  participation  w/in  enrolled  units 

Small  sample  sizes  that  reduce  confidence  in  findings 

Interrupted  access  to  ALD  products 

Demotivation,  participant  drop-outs,  loss  of  data 

Unpredictable  external  events 

Conflicting  priorities,  demotivation,  participant  attrition 

Participant-Related  Factors 

Shortage  of  time  due  to  OPTEMPO 

Inability  of  motivated  participants  to  use  ALD  tools  fully 

Positive  effects  of  pre-test  tools 

Difficulties  attributing  positive  effects  to  ALD  training 

Low-fidelity  product  utilization 

Skewed  performance  data  and  unreliable  user  feedback 

Incomplete  participation  by  Soldiers 

Incomplete  data  sets,  disproportionate  loss  of  later  measures 

Attrition  of  participants 

Loss  of  data  due  to  participants  departing  or  traveling 

Missed  data  collection  opportunities 

Data  gaps  such  as  missing  user  feedback  and  incident  logs 

The  limiting  faetors  listed  in  Table  8  are  fairly  typieal  of  field  investigations  eonducted 
within  operational  units.  The  pressures  of  today’s  COE  make  it  very  diffieult  for  tactical  units 
and  their  individual  Soldiers  to  dedicate  substantial  time  to  self-development  aetivities.  These 
pressures,  whieh  partly  motivated  the  ALD  program  in  the  first  place,  hampered  enrollment  of 
baseline  units  and  led  to  low  participation  rates  and  drop-outs  among  well-intentioned  Soldiers. 
The  data  colleetion  eonsequences — espeeially  the  relatively  small  sample  sizes,  incomplete  data, 
and  absence  of  post-training  measures — colleetively  lower  the  eonfidence  that  is  warranted  in  the 
data.  Significantly,  the  partieipation  by  only  AC  units  means  that  the  assessment  findings  eannot 
be  generalized  to  RC  units  without  extreme  caution. 

Program  of  Instruction 

The  ALD  “program  of  instruetion”  was  defined  chiefly  by  the  mix  of  tools,  the  target 
audience,  the  training  methods,  and  the  training  model.  This  seetion  discusses  these  factors, 
acknowledging  that  the  program  was  largely  a  collection  of  independent  tools  eonnected  loosely 
by  the  training  model. 
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Mix  of  Tools 


The  Detailed  Test  Plan  called  for  examining  the  completeness  and  sufficiency  of  the 
ALD  toolkit  (see  Appendix  A,  page  A-1 1,  in  Volume  II).  This  was  to  be  accomplished  at  the  end 
of  the  operational  demonstration  by  interviewing  leaders  of  participating  units  and  conducting 
focus  groups  with  ALD-experienced  participants.  However,  the  research  team  was  not  able  to 
collect  relevant  data  because  of  time  constraints  within  the  units  and  shortage  of  participants  with 
robust  experience  using  the  ALD  tools. 

As  explained  earlier,  the  toolkit  included  (a)  a  suite  of  diagnostic  instruments  to  boost 
self-awareness,  (b)  the  LASTS  modules  for  teaching  metacognitive  skills,  (c)  CTS  modules  for 
enhancing  cognitive  skills,  and  (d)  AXL  vignettes  for  building  interpersonal  skills.  The  unit 
participants  recognized  a  role  for  each  of  these  components  (see  Tables  9,  1 1  and  12),  but  they 
were  not  asked  to  identify  other  components  that  might  be  needed.  It  is  reasonable  to  conclude 
that  the  ALD  toolkit  contains  useful  components  but  is  not  necessarily  a  complete  family  of  tools 
for  the  full  spectrum  of  self-development  needs. 

The  end-of-program  user  satisfaction  survey  asked  the  unit  participants  to  write  in  which 
of  the  four  groups  of  ALD  tools  was  most  useful.  Among  the  modest  sub-sample  responding,  the 
responses  were  distributed  fairly  evenly  (Table  9).  Responses  from  participants  who  worked  with 
only  one  tool  were  excluded  from  the  table,  but  it  is  likely  that  some  responses  were  based  on 
working  with  only  two  or  three  of  the  tools.  Further,  a  participant  may  have  used  a  particular  tool 
to  a  limited  extent.  The  response  patterns  were  remarkably  similar  for  the  two  separate 
battalions.  These  results  indicate  no  notable  preference  for  one  ALD  component  over  another.  It 
is  tempting  to  infer  that  the  four  components  were  judged  to  be  equally  important,  but  equating 
usefulness  with  relative  or  absolute  importance  is  questionable.  Further,  the  results  must  be 
tempered  by  the  facts  that  the  sample  was  not  robust  and  some  participants  spent  too  little  time  to 
fully  appreciate  all  of  the  tools  (see  Tables  13  and  14).  The  participants  took  part  in  the  AXL 
vignettes  as  a  captive  audience  (guaranteeing  complete  exposure),  but  not  in  their  work  with  the 
other  tools. 

Table  9 

Frequency  of  Unit  Participants  ’  Responses  Regarding  Most  Useful  ALD  Tool 


Battalion 

Pre- 

Measures 

LASTS 

CTS 

AXL 

None 

1  (n  =  9) 

2 

2 

2 

2 

1 

2  (n  =  22) 

6 

6 

6 

4 

0 

All  (A/  =  31) 

8 

8 

8 

6 

1 

%  ol  All 

26% 

26% 

26% 

19% 

3% 

Unit  participants  were  also  asked  to  write  in  which  of  the  four  groups  of  ALD  tools  was 
least  useful  (Table  10).  The  most  frequent  response  was  the  diagnostic  pre -measures  (50%  of 
respondents),  and  next  came  the  AXL  vignettes  (19%).  Only  one  respondent  selected  the  LASTS 
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modules  as  the  least  useful.  Coming  from  the  self-seleeted  sample,  these  results  are  ineonsistent 
with  other  data  from  unit  partieipants  (see  espeeially  Table  31  below),  and  their  implieations  for 
the  ALD  toolkit  are  open  to  debate.  Although  we  didn’t  investigate  it,  different  tools  may  have 
been  judged  useful  depending  on  a  partieipant’s  status  (e.g.,  a  “young”  offieer  vs.  a  more 
seasoned  NCO). 

Table  10 

Frequency  of  Unit  Participants  ’  Responses  Regarding  Least  Useful  ALD  Tool 


Battalion 

Pre- 

Measures 

LASTS 

CTS 

AXL 

None 

All 

II 

1 

0 

0 

3 

2 

1 

2(n=  19) 

12 

1 

3 

2 

1 

0 

All  [N  =  26) 

13 

1 

3 

5 

3 

1 

%  of  All 

50% 

4% 

12% 

19% 

12% 

4% 

In  short,  the  ALD  toolkit  appears  to  eontain  a  sensible  mix  of  training  eapabilities  but  it 
eould  be  ineomplete.  Additional  researeh  would  be  required  to  determine  what  additional  tools 
might  be  needed,  if  any. 

Target  Audience 

By  design,  the  ALD  developers  seleeted  eompany  grade  offieers  as  the  primary  target 
audienee  for  the  family  of  tools,  with  platoon  and  eompany  NCOs  as  a  seeondary  audienee.  In 
this  projeet’s  operational  demonstration,  NCOs  were  ineluded  in  the  pool  of  eligible  leaders  at 
the  request  of  the  battalion  eommanders.  The  assessment- foeused  researeh  team  targeted  taetieal 
units  to  partieipate  in  the  demonstration,  beeause  they  expeeted  the  greatest  need  for  the  self¬ 
development  tools  there. 

Many  unit  partieipants  and  user  jury  members  indieated  that  the  target  audienee  should 
not  be  eompany  grade  offieers  in  operational  units.  Rather,  they  stated  the  best  use  for  the  ALD 
toolkit  lies  in  leader  preparation  arenas  sueh  as  the  U.S.  Military  Aeademy  (USMA)  or  NCO 
edueation  eourses  sueh  as  the  Advaneed  NCO  Course  (ANCOC).  The  reasons  for  targeting  these 
arenas  ineluded: 

•  Military  sehooling  “is  when  leadership  skills  and  traits  are  being  taught  and 
established.” 

•  Junior  offieers  assigned  to  units  have  already  reeeived  training  on  the  ALD  topies  and 
their  limited  time  should  be  spent  on  praetieal  leadership  experienee. 

•  Notably,  some  USMA  graduates  stated  that  they  had  already  reeeived  the  AXL 
training. 

When  the  unit  partieipants  were  asked  to  eheek  the  bloeks  of  formal  education  programs 
where  “any  of  this  training”  would  fit  best,  the  results  seen  in  Table  1 1  emerged.  The  patterns 
were  reasonably  similar  for  the  two  battalions.  Overall,  the  most  frequent  selection  was  the  Basic 
Officer  Leader  Course  (BOLC,  67%),  followed  by  the  USMA  or  Reserve  Officer  Training  Corps 
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(USMA/ROTC,  51%)  and  the  Basic  Noncommissioned  Officer  Course  (BNCOC,  38%).  In 
general,  the  response  frequencies  declined  for  the  more  advanced  courses  and  they  were  lower 
for  NCO  courses  than  for  comparable  commissioned  officer  programs.  Taken  as  a  whole,  these 
results  indicate  that  the  unit  participants  recognized  a  role  for  the  ALD  toolkit  in  institutional 
programs,  especially  for  junior  leaders  and  officer  cadets. 

Table  1 1 

Frequency  of  Unit  Participants  ’  Responses  Regarding  Best-Fit  Potential  of  ALD  Toolkit 


Battalion 

BNCOC 

ANCOC 

USMA/ 

ROTC 

BOLC 

CCC 

CGSC 

1  (n=  15) 

5 

5 

8 

10 

1 

0 

2  (n  =  30) 

12 

10 

15 

20 

10 

6 

All  [N  =  45) 

17 

15 

23 

30 

11 

6 

%  of  All* 

38% 

33% 

51% 

67% 

24% 

13% 

Note:  BNCOC  =  Basic  NCO  Course;  ANCOC  =  Advanced  NCO  Course;  USMA  =  U.S.  Military 
Academy;  ROTC  =  Reserve  Officer  Training  Corps;  BOLC  =  Basic  Officer  Leader  Course; 

CCC  =  Captains  Career  Course;  CGSC  =  Command  and  General  Staff  College 
*  Because  respondents  could  select  more  than  one  item,  percentages  total  more  than  100%. 

In  the  case  of  the  LASTS  user  juries,  the  user  satisfaction  survey  asked  the  Soldiers  to 
circle  the  military  education  programs  where  the  LASTS  modules  could  play  a  valuable  role. 
Table  12  summarizes  the  results,  which  reveal  quite  similar  patterns  across  the  four  different  user 
jury  groups.  In  all,  the  most  frequent  selection  was  USMA/ROTC  (84%),  followed  by  BOLC 
(38%)  and  self-development  (32%).  Two  respondents  selected  the  Sergeants  Major  Academy 
and  one  selected  CGSC.  In  addition,  three  Soldiers  wrote  in  Officer  Candidate  School.  The 
pattern  of  results  in  Table  12  is  similar  to  the  pattern  seen  in  Table  11,  including  the  decline  for 
more  advanced  courses/programs  and  the  less  frequent  selection  of  NCO  courses.  The  user  juries 
saw  a  clear  role  for  LASTS  training  in  institutional  programs. 

Table  12 

Frequency  of  User  Jury  Responses  Regarding  Potential  Role  of  LASTS 


Jury 

BNCOC 

ANCOC 

USMA/ 

ROTC 

BOLC 

CCC 

SELF- 

DVLPT 

1  (n=  15) 

3 

3 

14 

6 

2 

4 

2  (n  =  6) 

0 

0 

4 

1 

0 

3 

3  (n  =  9) 

2 

0 

8 

2 

0 

2 

II 

4 

2 

5 

5 

2 

3 

All  {N  =  37) 

9 

5 

31 

14 

4 

12 

%  of  All* 

24% 

14% 

84% 

38% 

11% 

32% 

*  Because  respondents  could  select  more  than  one  item,  percentages  total  more  than  100%. 
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Many  unit  participants  and  user  jury  members  stated  that  the  ALD  toolkit  would  be  very 
suitable  for  use  in  an  institutional  environment.  With  an  eye  to  enabling  greater  payoff  from  the 
tools,  the  Soldiers  mentioned  several  eonsiderations  regarding  institutional  use: 

•  Institutions  would  be  the  best  way  to  reach  the  population  most  likely  to  benefit  from 
ALD  training — eadets,  officer  candidates,  or  platoon/company  level  NCOs. 

•  Whereas  time  constraints  are  severe  in  tactieal  units,  time  for  working  with  ALD  tools 
would  be  available  in  an  institutional  setting. 

•  The  strong  requirements  for  equipment-specific  and  mission  essential  training  in 
tactical  units  would  not  be  a  factor  in  a  school  house  environment. 

•  In  the  institution  the  ability  of  students  to  diseuss  and/or  internet  with  instruetors  or 
small  groups  of  students  would  increase  the  effectiveness  of  the  ALD  tools. 

In  summary,  the  demonstration  partieipants’  feedbaek  pointed  to  a  different  target 
audienee  than  what  was  envisioned  originally.  Many  Soldiers  felt  that  the  early  stages  of  leader 
development  (eadet  status  for  offieers,  primary  leader  schooling  for  NCOs)  are  most  suitable  for 
ALD  training.  This  shifts  the  focus  from  operational  units  to  military  edueation  institutions. 

Training  Methods 

The  ALD  toolkit  relies  heavily  on  distributed  learning  methods  implemented  at  the 
individual  level.  These  bedroek  features  of  the  ALD  training  methods — eomputer  delivery  and 
individuals  working  alone — drew  considerable  feedbaek  from  the  demonstration  participants, 
although  quantitative  data  were  not  obtained. 

Not  surprising,  unit  participants  and  user  juries  both  expressed  the  belief  that  leadership 
training  is  essential.  However,  some  asserted  that  computer  based  training  is  not  well  suited  to 
building  leadership  skills.  In  one  user  jury,  members  stated  that  computer  delivered  training  has 
limited  benefit  because  it  is  just  “check  the  block.”  Since  the  Army  is  digitized  and  eompany 
grade  Soldiers  grew  up  with  computers,  such  opinions  do  not  likely  result  from  low  computer 
literacy  or  anti-computer  biases.  Rather,  the  distributed  learning  skeptics  mentioned  their  past 
experienee  with  computer  based  training  and  their  convietion  about  favorable  methods  for  leader 
development.  Their  preferred  methods  included  mentoring,  counseling,  discussion  with  peers 
and  supervisors,  and  practical  experience. 

The  concerns  about  the  suitability  of  Web-delivered  training  for  developing  leadership 
skills  could  work  against  acceptance  of  the  ALD  toolkit.  Even  with  today’s  emphasis  on 
distributed  learning  in  the  Army,  Soldiers  and  leaders  may  resist  computer-based  training.  If  the 
targeted  users  and  the  chain  of  command  are  not  eonvinced  of  the  value  and  effectiveness  of  the 
training,  it  will  not  be  given  a  chanee  to  demonstrate  a  positive  effect.  One  skeptical  battalion 
commander  summed  up  the  issue  by  asking,  “How  can  you  teach  interpersonal  skills  online?” 

When  many  Soldiers  stressed  mentoring,  counseling,  discussion,  and  practical  experience 
as  high-value  approaehes  for  effective  leadership  development,  they  usually  indicated  or  implied 
that  interpersonal  interaction  and  learning  by  doing  were  important  considerations.  The 
following  participant  comments  illustrate  the  beliefs  that  surfaeed: 

•  “A  computer  tool  cannot  take  the  place  of  human  interaction.” 
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•  “Face  to  face  feedback  is  more  important  than  online  feedback.” 

•  The  benefits  of  person-to-person  discussion  are  important;  .  it’s  difficult  to  ask  a 
computer  why  it  believes  your  answer  is  wrong.” 

•  You  can't  teach  leadership  in  a  class  because  “leaders  are  made.” 

•  “Adding  on  all  of  this  ‘cost  effective’  and  ‘high  tech’  leadership  training  is  simply 
taking  time  away  for  getting  actual  leadership  experience.” 

Related  to  the  emphasis  on  interpersonal  interaction  was  the  frequently  expressed  belief 
that  working  with  others — in  dyads  or  groups — is  essential  for  effective  leadership  training. 
Members  of  one  user  jury  felt  that  LASTS  training  would  benefit  from  small  group  discussion.  A 
member  of  another  user  jury  suggested  that  the  Army  needs  to  change  its  notion  that  self¬ 
development  is  a  solo  endeavor,  and  instead  place  critical  emphasis  on  social  learning.  The  target 
audience  beliefs  about  group  versus  individual  methods  for  training  leadership  skills  suggest  that 
special  effort  may  be  required  to  create  conditions  for  successful  fielding  and  implementation  of 
self-development  tools.  At  the  same  time,  it  may  be  worth  reexamining  design  principles  that 
ignore  interpersonal  interaction  as  an  element  of  self-development  tools. 

Given  the  demonstration  participants’  stated  beliefs  that  mentoring  and  counseling  are 
important,  a  related  dimension  of  the  ALD  training  methodology  merits  discussion.  To  counter 
the  absence  of  live  instructors  or  mentors,  the  developers  took  deliberate  steps  to  build  feedback 
into  some  of  the  tools.  For  example,  the  TKML  instrument  generates  feedback  based  on  expert 
ratings  of  the  courses  of  action  outlined  for  the  various  scenarios.  And  the  AXL  modules  include 
the  interactive  capability  to  query  the  actors  and  receive  scripted  commentary  about  their  thought 
process.  While  the  participants  were  not  asked  to  comment  directly  on  these  features,  the  proxy 
mentoring  may  have  contributed  to  the  positive  opinions  conveyed  in  the  user  satisfaction  survey 
data  (for  example,  see  Tables  15  and  21). 

Training  Model 

As  explained  in  the  Utilization  Model  section,  the  ALD  training  model  recommended  an 
optimal  sequence  for  the  various  tools.  This  sequence  began  with  self-awareness  diagnostics, 
then  proceeded  to  LASTS  training,  then  called  for  CTS  and  AXL  training  (intermixed),  and 
concluded  with  diagnostic  post-measures.  During  the  pre -training  diagnostics  and  LASTS 
modules,  the  LMS  enforced  the  recommended  sequence.  Past  the  LASTS  training,  the  LMS 
allowed  users  to  self-select  the  order  of  the  remaining  modules.  The  research  team  was  unable  to 
gather  user  feedback  on  the  training  model  because  there  were  too  few  participants  who  became 
fully  familiar  with  the  complete  toolkit.  Thus,  the  suitability  of  the  utilization  model  remains  an 
open  question. 

Soldiers  from  the  participating  battalions  occasionally  expressed  a  desire  for  latitude  to 
choose  their  own  sequence  of  working  with  the  ALD  tools.  This  appeared  to  reflect  a  sense  that 
the  participants  could  make  intelligent  decisions  about  how  to  invest  their  leader  development 
time.  There  was  also  an  implication  that  the  LMS-enforced  sequence  was  too  confining  and 
prevented  participants  from  browsing  the  various  tools  before  deciding  where  to  start.  Near  the 
end  of  the  participation  period  for  the  two  active  duty  battalions  the  research  team  decided  to 
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open  the  ALD  suite  to  self-sequencing,  but  the  expected  follow-on  units  failed  to  materialize.  In 
effect,  user  flexibility  in  sequencing  the  ALD  work  was  accepted  as  a  valuable  feature. 

Occasionally  Soldiers  stated  that  the  ALD  toolkit  should  acknowledge  other  leader 
development  programs.  Such  programs  include  the  self-appraisal  features  of  the  current  Officer 
Evaluation  Report,  leadership  coursework  incorporated  in  USMA  and  ROTC  curricula,  the 
Center  for  Army  Leadership’s  Multi- Source  Assessment  and  Feedback  program,  and  a  handful 
of  tools  available  through  the  Army  Knowledge  Online  (AKO)  Web  site.  Members  of  one  user 
jury  suggested  that  the  LASTS  tool  should  be  hosted  through  AKO  so  other  leader  development 
tools  would  be  readily  available.  Soldiers  in  another  user  jury  stated  that  the  LASTS  modules 
should  incorporate  Army  leadership  references  such  as  AR  600-100,  Army  Leadership  (U.S. 
Department  of  the  Army,  2007),  FM  6-22,  Army  Leadership  (U.S.  Department  of  the  Army, 
2006)  and  DA  Pamphlet  350-58,  Leader  Development  for  America ’s  Army  (U.S.  Department  of 
the  Army,  1994).  The  ALD  program  does  not  exist  in  a  vacuum,  and  pointing  to  other  programs 
and  references  would  reinforce  the  bigger  picture. 

Learning  Environment 


Distributed  Learning  Aspects 

The  ALD  Web  site  was  hosted  on  a  government  server  with  the  Moodle  LMS  installed. 
No  special  software  on  the  user’s  computer  was  required,  aside  from  the  requisite  version  of  the 
software  for  displaying  multimedia  files.  The  user  authentication  process  that  required  Soldiers 
to  set  up  an  account  drew  no  criticism  from  participants,  but  a  few  questioned  why  the  Web  site 
did  not  rely  on  AKO’s  single  sign-on  capability. 

The  24x7  internet  access  to  the  ALD  Web  site  was  maintained  without  incident.  If  any 
participants  encountered  bottlenecks  in  their  internet  access,  it  did  not  come  to  the  attention  of 
the  research  team.  Few  technical  issues  and  problems  were  reported  by  the  users,  although  one 
participant  stated  that  his  first  two  attempts  to  login  failed,  leading  him  to  stop  trying. 

Relying  on  commercially  available  Web  browsers  as  the  software  framework  for  the  user 
interface  provided  a  familiar  look  and  feel  to  the  target  audience.  The  users  made  no  negative 
comments  about  the  appearance  or  functionality  of  the  interface.  Presumably  they  found  the 
explanatory  tags  appended  to  each  main  menu  link — such  as  “Fearn  about  your  own  leadership 
style” — to  be  useful.  The  requirement  to  enable  the  display  of  pop-ups  such  as  graphic  objects 
apparently  did  not  pose  a  problem  for  any  of  the  participants.  The  success  of  the  user  interface 
most  likely  stemmed,  in  part,  from  the  pilot  testing  and  the  research  team’s  efforts  spent  making 
minor  improvements  during  the  early  stages  of  the  operational  demonstration. 

More  detailed  findings  on  technical  aspects  of  the  ALD  Web  site’s  delivery  environment 
can  be  found  in  the  companion  report  by  Woller  et  al.  (in  preparation). 
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Equipment  Availability 

On-site  visits  by  research  team  members  revealed  that  computers  were  generally  not 
available  below  the  company  headquarters.  This  was  true  of  both  battalions  participating  in  the 
operational  demonstration.  However,  only  one  unit  participant  commented  that  the  limited  access 
to  unit  computers  was  a  problem  for  using  the  online  ALD  tools.  The  absence  of  participant 
comments  might  result  from  Soldiers  taking  the  issue  for  granted.  Another  possible  explanation 
is  the  tendency  of  Soldiers  to  overcome  problems  to  accomplish  assigned  tasks. 

Regardless,  the  participating  units  had  few  computers  and  they  were  most  likely 
dedicated  to  administrative  functions  such  as  personnel  and  supply  management.  If  the  ALD 
program  is  to  be  effective,  there  must  be  sufficient  access  to  unit  computers  along  with  internet 
access.  It  is  not  known  how  many  participants  used  their  own  computers  to  access  the  ALD  Web 
site.  However,  occasional  comments  indicated  that  this  was  an  unwelcome  option  because  it 
extended  long  duty  days  and  infringed  on  “Family  time.” 

Operational  Constraints 

A  strong  recurring  theme  in  comments  from  both  unit  participants  and  user  juries  was  the 
compelling  impact  of  the  COE  on  the  professional  development  of  junior  leaders,  both  officers 
and  NCOs,  in  tactical  units.  Four  aspects  of  the  COE  became  clear: 

•  Eieutenants  and  NCOs  are  being  promoted  quickly  and  given  increased 
responsibilities  when  they  may  have  insufficient  experience. 

•  Although  some  units  commit  time  to  leadership  development  training,  more  effort  and 
opportunities  are  needed. 

•  The  high  OPTEMPO  in  units,  regardless  of  the  lifecycle  phase,  adversely  impacts 
leadership  development.  Junior  officers  and  NCOs  have  more  demands  on  their  time 
than  can  be  met,  even  with  long  days. 

•  Requirements  placed  in  the  unit  training  schedule  define  priorities.  As  one  participant 
stated,  ‘The  plate  is  full  and  anything  that  is  not  a  requirement  falls  off  of  the  plate.” 

The  unit  participants  and  user  juries  asserted  that,  to  be  effective,  time  must  be  dedicated 
to  using  the  ALD  tools.  Expecting  Soldiers  to  “make  time"  for  the  bulk  of  self-development 
training  will  not  work.  Participants  noted  (a)  the  duties  and  training  that  are  scheduled  during 
unit  reset,  and  (b)  the  “difficulty  fitting  the  training  into  workdays  which  are  already  12  hours 
long.”  In  conjunction  with  the  long  hours  and  fast  pace,  three  other  aspects  surfaced: 

•  Using  the  ALD  toolkit  at  home  conflicts  with  being  “a  Family  man/Soldier,”  making  it 
hard  to  find  time  for  AED  work  outside  of  duty  hours. 

•  A  few  participants  thought  that  ALD  would  interfere  with  more  valuable  training.  For 
example,  “My  Soldiers  have  not  been  able  to  train  because  of  my  time  here.” 

•  Some  felt  that  ALD  “training  was  both  valuable  and  relevant”  but  the  time  required 
was  a  “distraction  from  scheduled  training.” 

One  leader  summed  up  the  consensus  of  unit  participants:  “Scheduling  this  [ALD] 
training  and  blocking  time  on  the  training  schedule  is  the  only  effective  way  of  executing  this 
event  in  an  MTOE  [Modified  Table  of  Organization  and  Equipment]  unit.” 
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Online  Usage  of  ALD  Tools 


A  total  of  66  participants  logged  into  the  ALD  Web  site,  eventually  aeeumulating  an 
average  of  14.0  logins  (range,  1-50).  After  ereating  an  aeeount,  participants  had  the  choiee  of 
viewing  the  Student  Guide  or  directly  entering  the  pre -measures  seetion  of  the  Web  site,  an  area 
where  the  LMS  reeorded  start/stop  times  for  most  of  the  instruments.  A  total  of  62  partieipants 
entered  one  or  more  time-reeording  areas  of  the  ALD  Web  site  at  least  onee. 

Diagnostic  Pre-Measures 

The  diagnostie  pre-measures  were  the  first  tools  aeeessed  by  the  partieipants.  Table  13 
lists  the  instruments  in  the  order  in  whieh  a  partieipant  eneountered  the  links  in  the  main  menu. 
As  the  table  shows,  different  numbers  of  participants  used  the  diagnostie  instruments  to  varying 
degrees.  Nearly  all  partieipants  who  viewed  an  instrument  spent  some  time  responding  to  the 
questions,  but  many  ehose  to  exit  before  reaehing  the  end  of  the  tools.  The  frequeney  eounts  in 
the  table  suggest  an  order  effect,  such  that  usage  of  the  tools  deelined  aeross  the  sequenee  of 
instruments  from  first  to  last,  exeept  for  the  Personality  Assessment. 

Table  13 

Unit  Participants  ’  Usage  of  Pre-Measures,  by  Instrument 


Pre-Measure  Instrument 

# 

Viewed 

# 

Compl’d 

Viewing  Time  (min) 

Median 

Range 

Demographic  Survey 

61 

53 

N/C* 

N/C* 

Unit  Cohesion  Index 

59 

56 

1.3 

1.2-16.3 

Leadership  Experiences  Survey 

57 

36 

11.9 

1.3-49.2 

Multifactor  Leadership  Questionnaire 

48 

48 

N/C* 

N/C* 

Leadership  Opinions  Survey 

49 

48 

9.6 

3.2-69.6 

Tacit  Knowledge  for  Military  Leaders  -  Platoon  Level 

24 

23 

20.0 

4.2-60.0 

Tacit  Knowledge  for  Military  Leaders  -  Company  Level 

24 

22 

24.9 

0.02-60.2 

Personality  Assessment 

44 

33 

N/C* 

N/C* 

*  N/C  =  not  captured. 


The  time  spent  on  the  various  instruments  varied  eonsiderably,  depending  largely  on  the 
number  and  eomplexity  of  the  items  eomprising  the  measures.  As  Table  13  shows,  the  UCI  took 
the  least  time  (median,  1 .3  min)  while  the  eompany-level  version  of  the  TKML  instrument  took 
the  longest  time  (median,  24.9  min).  The  times  for  the  Leadership  Opinions  and  Leadership 
Experienees  surveys  fell  in  between  (medians  of  9.6  and  11.9  min,  respeetively).  Oeeasional 
partieipants  spent  more  than  an  hour  on  the  lengthier  diagnostie  instruments. 

Training  Modules 

After  completing  the  diagnostie  pre-measure  instruments,  partieipants  proeeeded  to  the 
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training  section  of  the  Web  site.  The  available  training  tools  included  the  LASTS  modules  and 
the  CTS  modules  (the  AXL  vignettes  were  not  available  online). 

•  At  least  one  LASTS  module  was  viewed  by  41  participants,  with  a  range  of  1-7  viewings 
per  participant  and  a  mean  of  1 .63.  No  time  on  task  data  were  recorded. 

•  The  CTS  suite  contained  8  modules,  each  with  3  components — pre-test,  training  lessons, 
and  post-test.  Participants  could  access  any  component  of  the  CTS  modules  for  any 
length  of  time  at  their  discretion.  Discussion  of  CTS  usage  data  follows. 

Thirty  participants  accessed  the  CTS  menu  option  and  viewed  some  component  of  the 
CTS  modules  at  least  once,  with  the  LMS  recording  start  and  stop  times.  Because  they  could 
choose  their  own  sequence  of  modules  and  components,  participants  often  had  gaps  in  their  CTS 
usage  data.  For  example,  a  participant  may  have  taken  the  pre-test  for  Module  3  but  not  the  post¬ 
test,  or  vice  versa.  Also,  participants  who  took  pre-  and/or  post-tests  did  not  always  access  the 
training  portions  of  the  modules.  Those  who  did  access  an  actual  lesson  did  not  necessarily 
undergo  training,  since  the  time  they  spent  on  the  lesson  was  sometimes  too  scant  (occasionally 
less  than  a  minute)  for  reading  or  studying  the  material. 

As  seen  in  Table  14,  participant  usage — especially  time  on  task — varied  widely  for  any 
given  CTS  component.  The  patterns  across  modules  show  an  order  effect,  with  the  number  of 
participants  declining  for  modules  coming  later  in  the  CTS  sequence  especially  when  comparing 
modules  1-4  with  modules  5-8.  This  is  true  for  all  three  components — pre-test,  training,  and 
post-test.  Modules  5-8  drew  participants  rarely.  The  order  effect  is  also  evident  in  the  time  on 
task  data,  especially  for  training  and  post-test  components. 

Table  14 

Time  (Min)  Spent  by  Unit  Participants  Working  on  CTS  Modules,  by  Component 


Module 

Pre-Test 

Training 

Post-Test 

n 

Mean 

Range 

n 

Mean 

Range 

n 

Mean 

Range 

1 

16 

9.3 

1.1-22.3 

8 

26.5 

0.3-87.6 

22 

21.3 

3.1-39.3 

2 

13 

15.2 

3.7-30.3 

7 

5.1 

0. 1-9.3 

20 

21.4 

1.9-36.7 

3 

13 

20.0 

3.1-51.0 

8 

8.2 

0.2-29.0 

18 

8.8 

0.9-21.4 

4 

11 

16.3 

6.1-27.5 

9 

14.4 

0.1-54.2 

18 

16.4 

2.6-29.6 

5 

1 

N/A® 

(6.2) 

1 

N/A® 

(0.3) 

5 

8.9 

1.0-14.8 

6 

1 

N/A® 

(20.1) 

1 

N/A® 

(1.4) 

5 

5.0 

1. 1-9.7 

7 

1 

N/A® 

(3.8) 

0 

N/A 

N/A 

5 

5.4 

1.2-10.8 

8 

0 

N/A 

N/A 

0 

N/A 

N/A 

5 

5.2 

o 

00 

®  The  sample  size  (n  =  1)  did  not  support  computing  a  mean. 


The  training  component  of  each  CTS  module  consisted  of  interactive  lessons  on  the 
designated  skill  and  sub-skills  (see  The  Leader  Development  Toolkit  section).  Only  1 1  of  the  30 
participants  who  accessed  the  CTS  tool  spent  time  viewing  the  training  portion  of  any  module. 
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and  the  times  were  generally  insuffieient  to  expeet  substantive  learning  to  oeeur.  As  the  ranges  in 
Table  14  show,  the  viewing  time  for  the  training  modules  frequently  fell  below  1.5  min,  whieh  is 
negligible.  The  mean  training  times  for  Modules  2  and  3  were  under  9  min,  while  the  highest 
mean  viewing  time  was  26.5  min.  This  is  notable  eonsidering  that  eaeh  module  was  designed  to 
engage  the  learner  for  approximately  2  hr.  This  indieates  that  for  the  seven  modules  viewed, 
most  of  the  partieipants  did  not  spend  enough  time  for  learning  to  oeeur.  At  the  same  time,  some 
partieipants  did  spend  substantial  time  (as  mueh  as  87  min)  on  seleeted  modules. 

Post-tests  were  the  most  frequently  viewed  CTS  eomponent.  The  time  spent  viewing  eaeh 
post-test  varied  widely  from  under  1  min  to  over  39  min.  It  is  not  elear  why  the  post-tests 
appeared  to  draw  more  attention  from  the  partieipants  than  the  pre-tests  or  training  lessons,  but 
some  Soldiers  may  have  eonsidered  the  post-test  an  opportunity  to  demonstrate  their  pre-existing 
knowledge,  a  means  of  getting  eredit  without  taking  the  lessons,  or  a  personal  ehallenge.  It  is 
also  possible  that  some  expeeted  a  post-test  would  give  a  quiek  overview  of  the  instruetional 
eontents,  presuming  that  the  pre-test  would  serve  an  attention-getting  or  motivational  purpose. 

Beeause  AXL  training  was  eondueted  offline,  no  usage  data  were  eolleeted  by  the  ALD 
Web  site’s  LMS.  The  first  AXL  vignette  oeeurred  1-2  months  after  the  start  of  ALD  training, 
while  the  seeond  vignette  took  plaee  near  the  end  of  the  ALD  training  window.  Appearing  on  the 
unit  training  sehedule,  the  vignette  sessions  drew  partieipants  in  groups  of  8-12  Soldiers  eaeh.  By 
the  end  of  the  ALD  operational  demonstration,  more  than  90  Soldiers  had  partieipated  in  one  or 
two  vignette  sessions. 

Diagnostic  Post-Measures 

At  the  end  of  ALD  training,  partieipants  were  asked  to  eomplete  the  diagnostie  post¬ 
measures.  Intended  to  enable  pre-post  measurement  of  ALD  learning  effeets,  these  instruments 
ineluded  the  UCI,  MLQ  (alternate  version).  Leadership  Opinions  survey,  and  TKML  (alternate 
version).  Also  available  were  the  post-tests  from  all  eight  CTS  modules  to  evaluate  retention  of 
eritieal  thinking  knowledge  and  skills.  Only  three  partieipants  spent  time  on  the  post-measures, 
yielding  insuffieient  data  to  eharaeterize  the  usage  patterns. 

Suitability  of  the  Tools 


Diagnostic  Tools 

The  user  satisfaetion  survey  (see  Appendix  D,  page  D-6,  in  Volume  II)  ineluded  items 
addressing  the  measurement  aspeets  of  the  online  diagnostie  instruments.  For  example,  one  of 
the  questions  asked  the  partieipants  to  indieate  the  extent  to  whieh  the  MLQ  eovered  all  aspeets 
of  leadership.  The  researeh  team  administered  the  survey  to  the  unit  partieipants  present  at  the 
end  of  the  seeond  AXL  vignette  session — up  to  5  months  after  they  were  seheduled  to  eomplete 
the  pre-training  diagnosties. 

A  summary  of  the  survey-based  agree/disagree  rating  data  appears  in  Table  15.  The 
results  indieate  weakly  or  moderately  favorable  opinions  aeross  the  board,  with  more  than  40% 
of  the  respondents  agreeing  (slightly,  somewhat,  or  strongly)  with  the  positive  statements 
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regarding  the  various  tools.  Among  the  more  favorable  opinions  (66-74%  of  respondents 
agreeing)  were  the  TKML,  Leadership  Opinions,  Personality  Inventory,  and  UCI  instruments. 
With  only  one  exeeption,  the  proportion  of  the  respondents  registering  negative  opinions  was 
less  than  20%.  Overall,  these  results  indieate  that  the  diagnostie  tools  measure  relevant  and 
instructive  dimensions  of  leadership  in  the  eyes  of  Soldiers,  but  there  is  room  for  improvement. 
It  is  possible  that  the  ratings  were  influenced  by  more  than  the  stated  measurement  aspects  (e.g., 
time  required  for  completion). 

Table  15 

Unit  Participants’  Ratings  of  Measurement  Aspects  of  Pre-Training  Diagnostics  (N  =  34) 


Test  /  Measurement  Aspect 

Mean 

Rating* 

% 

Agreeing 

%  Dis¬ 
agreeing 

The  MLQ  questions  accurately  reflected  my  leadership  style 

4.6 

53% 

12% 

The  MLQ  questions  covered  all  aspects  of  leadership 

4.4 

41% 

24% 

The  MLQ  questions  accurately  described  me  as  a  leader 

4.5 

50% 

18% 

The  MLQ  reading  list  added  to  my  leadership  understanding 

4.7 

59% 

9% 

The  TKML’s  feedback  comparing  my  answers  to  SMEs  was  useful 

4.6 

68% 

15% 

The  TKML  reading  list  gave  me  some  useful  Information 

5.0 

71% 

3% 

The  Leadership  Qpinlons  questions  accurately  reflected  me  as  a  leader 

5.1 

74% 

6% 

The  Leader  Experiences  tool  captured  all  my  leadership  experiences 

4.6 

53% 

18% 

The  personality  Inventory  feedback  gave  me  personal  Insights 

4.8 

68% 

9% 

The  UCI  questions  were  pertinent  to  the  topic 

4.9 

68% 

3% 

The  UCI  allowed  me  to  correctly  express  the  attitudes  of  my  unit 

5.0 

66% 

6% 

*  The  rating  scale  ranged  from  1  (strongly  disagree)  to  7  (strongly  agree),  with  4  being  neutral. 


Of  the  six  online  diagnostic  instruments  used  to  generate  pre -training  self-awareness, 
three  merited  special  interest  because  of  their  aggregate  scoring  options  and  expected  sensitivity 
to  self-development  training  effects.  In  addition,  two  instruments  were  added  as  hardcopy 
(offline)  diagnostic  tools  and  were  administered  to  AXL  participants  as  baseline  measures.  The 
resulting  instruments  will  be  examined  in  detail  in  the  following  paragraphs: 

•  Self-awareness  and  pre-post  measures 

o  Multifactor  Leadership  Questionnaire  (MLQ) 
o  Tacit  Knowledge  for  Military  Leaders  (TKML) 
o  Unit  Cohesion  Index  (UCI) 

•  Baseline  measures 

o  Need  for  Cognitive  Closure  Survey 
o  Learning  Strategies  Survey 

The  MLQ  instrument  measured  leadership  style  by  asking  the  respondents  to  judge  how 
often  a  given  statement  (e.g.,  “I  spend  time  coaching”)  fit  them  as  leaders.  The  higher  the 
frequency  rating,  the  more  a  respondent  perceived  that  a  given  statement  applied  to  him.  The  45 
items  of  the  instrument  were  used  to  compute  rating-based  scores  for  13  subscales,  as  seen  in 
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Table  16.  For  the  sample  of  unit  partieipants,  the  table  gives  summary  deseriptive  statisties  for 
eaeh  subseale.  The  relationships  among  the  mean,  median,  and  minimum/maximum  values  for 
eaeh  subseale  indieate  basieally  normal  distributions.  Average  seores  varied  moderately  aeross 
the  subseales,  ranging  from  a  low  of  1.8  to  a  high  of  4.6.  For  the  majority  of  the  subseales,  mean 
seores  below  2.5  indieated  that  the  leadership  eharaeteristies  fit  the  partieipants  “sometimes.” 
There  are  insuffieient  post- test  data  against  whieh  to  eompare  the  results  in  Table  16. 


Table  16 

Unit  Participants  ’  Pre-Training  Scores  for  MLQ  Scales  and  Subscales  (N  =  48) 


Subscale 

Mean 

Rating* 

Median 

Rating 

Range 

Std  Dev 

Transactional  Leadership** 

2.4 

2.4 

1.38-3.12 

.47 

Transformational  Leadership** 

1.9 

1.9 

1. 0-3.2 

.56 

Inspirational  Leadership 

2.1 

2.0 

1.0-3.75 

.64 

Idealized  Influence 

2.0 

2.0 

1. 0-3.5 

.55 

Contingency  Reward 

2.0 

2.0 

1. 0-3.0 

.55 

Intellectual  Stimulation 

2.0 

2.0 

1. 0-3.5 

.44 

Individual  Consideration 

1.9 

1.9 

1. 0-3.0 

.46 

Management  by  Exception  (Active) 

2.8 

2.8 

1.5-4.25 

.67 

Management  by  Exception  (Passive) 

3.8 

3.8 

2.5-5.0 

.39 

Passive  Avoidance 

4.2 

4.2 

3.25-5.0 

.52 

Laissez-faire** 

4.6 

4.8 

3.75-5.0 

.40 

Attributed  Charisma  (N  =  42) 

1.8 

1.8 

1. 0-3.0 

.51 

Extra  Effort  (N  =  37) 

2.0 

2.0 

1. 0-3.0 

.55 

*  The  underlying  rating  scale  ranged  from  0  (not  at  all)  to  4  (frequently),  with  2  being  sometimes. 
**  Denotes  key  scales. 


The  platoon-level  TKML  test  involved  eight  seenario-based  ehallenges  with  a  total  of  81 
individual  ratings  of  different  eourses  of  aetion.  The  eompany-level  test  involved  ten  seenario- 
based  ehallenges  with  a  total  of  90  different  eourses  of  aetion  rated.  The  differenee  between  the 
experts’  average  rating  and  the  partieipant’s  rating  was  eomputed  for  eaeh  item,  and  the  absolute 
value  of  the  differenee  was  summed  aeross  all  items  to  produee  a  total  seore  indieating  the 
degree  of  varianee  from  expert  judgment.  The  lower  the  total  seore,  the  better  the  agreement  with 
expert  judgment.  Table  17  summarizes  the  resulting  data  for  the  pre-training  TKML  test.  The 
pattern  of  values  for  mean,  median,  and  minimum/maximum  seores  indieates  reasonably  normal 
distributions  for  both  TKML  versions.  The  average  performanee  on  the  tests  (parallel  versions) 
was  quite  similar  for  the  platoon-level  and  eompany-level  samples. 
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Table  17 

Unit  Participants’  Pre-Training  Scores  for  TKML  (Overall),  by  Echelon 


Echelon 

Mean 

Median 

Range 

Std  Dev 

TKML  Platoon  Level  (N  =  23) 

139.0 

138.6 

97.3-198.9 

24.6 

TKML  Company  Level  (N  =  22) 

145.5 

138.3 

110.3-208.7 

26.9 

Note:  The  scores  indicate  degree  of  variance  from  expert  judgment,  aggregated  across  items. 


The  Need  for  Cognitive  Closure  instrument  measured  four  aspeets  of  eognitive  style,  as 
seen  in  Table  18.  This  diagnostie  tool  asked  the  respondents  to  rate  how  mueh  they  agreed  or 
disagreed  with  a  given  statement — for  example,  “I  dislike  unpredietable  situations.”  The  35 
items  of  the  instrument  were  used  to  eompute  rating-based  seores  for  the  four  subseales.  For  the 
sample  of  unit  partieipants.  Table  18  gives  summary  deseriptive  statisties  for  eaeh  subseale.  The 
strong  agreement  between  mean  and  median  values  suggests  reasonably  normal  distributions. 
The  average  seores  vary  moderately  between  the  subseales,  ranging  from  a  low  of  3. 14  to  a  high 
of  4.41.  There  are  no  baseline  or  post-training  data  against  whieh  to  eompare  these  values. 

Table  18 

Unit  Participants  ’  Pre-Training  Scores  on  Need  for  Cognitive  Closure,  by  Subscale  (N  =  67) 


Subscale 

Mean 

Median 

Std  Dev 

Need  for  Cognitive  Closure  -  Predictability 

3.99 

4.00 

.82 

Need  for  Cognitive  Closure  -  Ambiguity 

4.03 

4.00 

.58 

Need  for  Cognitive  Closure  -  Closed-Mindedness 

3.14 

3.12 

.60 

Need  for  Cognitive  Closure  -  Order 

4.41 

4.60 

.72 

Note:  The  rating  scale  ranged  from  1  (strongly  disagree)  to  6  (strongly  agree),  with  no  neutral 
point  or  “don’t  know”  option. 

The  Learning  Strategies  instrument  measured  three  eognitive  proeess  aspeets  of  learning 
in  eomputer-based  training  environments,  as  seen  in  Table  19.  This  diagnostie  tool  asked  the 
respondents  to  rate  how  mueh  they  agreed  or  disagreed  with  a  given  statement  sueh  as  “I  think 
about  possible  alternatives.”  The  15  items  of  the  instrument  fed  the  ealeulation  of  rating-based 
seores  for  the  three  subseales.  For  the  sample  of  unit  partieipants.  Table  19  gives  summary 
deseriptive  statisties  for  eaeh  subseale.  Here  again,  the  strong  agreement  between  mean  and 
median  values  suggests  reasonably  normal  distributions  for  the  various  subseales.  There  is  little 
variability  among  the  means  for  the  three  subseales,  the  differenee  between  the  lowest  and 
highest  means  representing  about  one-third  of  the  typieal  standard  deviation. 
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Table  19 

Unit  Participants  ’  Pre-Training  Scores  for  Learning  Strategies  Subscales  (N  =  69) 


Subscale 

Mean 

Median 

Std  Dev 

Effort  regulation  in  computer  learning  environments 

3.74 

3.75 

.50 

Critical  thinking  in  computer  learning  environments 

3.97 

4.00 

.64 

Elaboration  in  computer  learning  environments 

3.93 

4.00 

.93 

Note:  The  underlying  rating  scale  ranged  from  1  (strongly  disagree)  to  6  (strongly  agree),  with 
neither  neutral  point  nor  “don’t  know”  option. 

As  Table  20  shows,  the  UCI  survey  yielded  measures  on  ten  dimensions  ranging  from 
trust  and  earing  among  Soldiers  and  leaders  to  teamwork  to  pride  in  unit  membership.  The  table 
ineludes  the  mean  rating  for  eaeh  dimension,  along  with  the  pereent  of  the  sample  agreeing  or 
disagreeing  that  the  eharaeteristie  was  true  for  their  unit.  “Unable  to  judge”  responses  have  been 
removed  from  the  analysis.  The  highest  mean  rating  was  3.2,  indieating  weak  agreement  with 
positive  statements  about  unit  eohesion,  at  best.  The  highest  proportion  of  the  sample  agreeing 
(ineluding  strongly  agreeing)  with  any  eharaeteristie  was  39%,  with  about  50%  of  the  sample 
ehoosing  “neither  agree  nor  disagree.”  The  highest  proportion  of  the  sample  disagreeing 
(ineluding  strongly  disagreeing)  with  any  eharaeteristie  was  52%,  with  about  one -third  of  the 
sample  ehoosing  “neither.”  The  pattern  of  mean  ratings  suggests  that  the  respondents 
diseriminated  among  the  various  dimensions  of  unit  eohesion,  but  generally  held  a  neutral  view 
of  eohesion  in  their  units. 


Table  20 

Unit  Participants  ’  Pre-Training  Ratings  for  Unit  Cohesion  Index  Dimensions  (N  =  56) 


Cohesion  Dimension 

Mean 

Rating* 

% 

Agreeing 

%  Dis¬ 
agreeing 

Members  and  leaders  support  Army  values  (Army  Values) 

3.2 

39% 

9% 

Members  trust  and  care  about  each  other  (Member  Bonding) 

3.0 

26% 

13% 

Members  work  together  as  a  team  to  get  the  job  done  (Teamwork) 

3.1 

38% 

11% 

Leaders  trust  and  care  about  each  other  (Leader  Bonding) 

3.1 

31% 

13% 

Leaders  care  about  and  heip  Soidiers  (Leader  Caring) 

3.1 

39% 

16% 

Leaders  have  skills  and  abilities  to  lead  in  combat  (Leader  Skills) 

2.9 

28% 

28% 

Members  know  what  is  expected  and  what  causes  trouble  (Rule  Clarity) 

3.1 

38% 

14% 

Members  feel  a  sense  of  pride  and  importance  in  the  unit  (Unit  Pride) 

2.9 

25% 

27% 

Members  feel  their  Family  and  social  needs  are  met  (Member  Needs) 

2.4 

11% 

52% 

Members  are  progressing  toward  their  goals  (Member  Goals) 

3.2 

39% 

12% 

*  The  rating  scale  ranged  from  1  (strongly  disagree)  to  5  (strongly  agree),  with  3  being  neither. 
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Training  Tools 


The  first  ALD  training  tool  encountered  by  the  unit  participants  was  the  LASTS  suite 
containing  five  primary  modules.  As  part  of  the  user  satisfaction  survey  administered  near  the 
end  of  unit  participation,  32  leaders  rated  various  aspects  of  the  LASTS  modules,  as  seen  in 
Table  21.  The  summary  of  results  in  the  table  indicates  moderately  favorable  opinions  of  the 
acceptability  of  the  modules.  The  mean  ratings  were  all  less  than  5.0  (equating  to  “slightly 
agree”),  with  44-65%  agreeing  (slightly,  somewhat,  or  strongly)  with  the  survey’s  statements. 
Between  28%  and  50%  of  the  respondents  were  “fence  sitters,”  choosing  “neutral”  as  their 
response  for  a  given  item.  Altogether,  these  results  indicate  that  the  LASTS  tool  as  a  whole  is 
acceptable  to  Soldiers,  with  substantial  room  for  improvement.  The  unit  participant  data  shed  no 
light  on  the  individual  modules  within  the  LASTS  suite  because  the  survey  addressed  the  tool 
only  globally. 

Four  separate  user  juries  examined  portions  of  the  LASTS  suite  in  a  group  setting  with 
shared  comments  and  discussion.  After  viewing  a  LASTS-wide  overview  and  then  a  selected 
module,  each  group  individually  completed  a  user  satisfaction  survey  tailored  to  the  specific 
module.  Thus  these  Soldiers  provided  ratings  for  the  specific  module  that  they  reviewed 
primarily  as  observers  rather  than  “participants.”  The  extent  to  which  they  were  able  to  imagine 
working  with  the  module  in  individual  fashion  is  unknown. 

Table  21 

Unit  Participants  ’  Ratings  of  Training  Aspects  of  ALD  Toolkit,  by  Training  Tool 


Training  Aspect 

Mean 

Rating* 

% 

Agreeing 

%  Dis¬ 
agreeing 

LASTS  Modules  (N  =  32) 

The  modules  appeared  easy  to  use 

4.7 

47% 

3% 

The  modules  related  directly  to  me  and  my  career 

4.6 

44% 

9% 

The  exercises  focused  on  realistic  situations  and  challenges 

4.9 

65% 

3% 

The  training  exercises  were  challenging 

4.5 

50% 

16% 

The  feedback  from  the  exercises  was  a  valuable  teaching  tool 

4.8 

63% 

9% 

The  modules  took  too  much  time 

4.8 

52% 

13% 

CTS  Modules  (N  =  33) 

The  training  exercises  were  challenging 

4.7 

52% 

9% 

The  feedback  from  the  exercises  was  a  valuable  teaching  tool 

4.9 

63% 

6% 

AXL  Vignettes  (N  =  40) 

The  films  focused  on  real  world  situations  &  leadership  problems 

5.0 

68% 

18% 

The  films  highlighted  critical  leadership  issues 

5.5 

80% 

8% 

Feedback  from  actors  revealed  leaders’  thought  processes 

5.1 

70% 

10% 

Group  discussion/feedback  provided  different  points  of  view 

5.7 

78% 

3% 

*  The  rating  scale  ranged  from  1  (strongly  disagree)  to  7  (strongly  agree),  with  4  being  neutral. 
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The  mean  user  jury  ratings  of  the  seleeted  aspeets  of  the  four  different  LASTS  modules 
appear  in  Table  22.  Beeause  the  rating  seale  had  no  neutral  point,  a  “middle  of  the  road”  average 
would  be  3.5  (rather  than  4.0  for  the  data  in  Table  21).  Unlike  the  results  from  the  unit 
partieipants,  the  user  jury  means  tend  to  re  fleet  slightly  to  moderately  negative  opinions  about 
the  LASTS  modules.  Four  dimensions  drew  moderately  unfavorable  ratings — eareer  relatedness, 
exereise  realism,  exereise  ehallenge,  and  feedbaek  value — with  grand  means  from  1.8  to  2.6 
(averaged  aeross  modules).  Notably,  the  Self  and  Soeial  Awareness  module  reeeived  an  overall 
rating  (averaged  aeross  aspeets)  that  was  moderately  unfavorable  (grand  mean,  1.9).  This  may 
refleet  that  group’s  eomments  eritieizing  the  presentation  of  information  in  the  module, 
espeeially  the  laek  of  interaetivity  and  low-quality  materials.  It  should  be  remembered  that  the 
user  jury  members  were  unable  to  judge  the  aeeeptability  of  the  modules  in  light  of  any  post¬ 
training  applieation  experienee. 

For  the  CTS  modules,  the  unit  partieipants’  ratings  of  two  training  aspeets  are  ineluded  in 
Table  21.  The  results  in  the  table  indieate  moderately  favorable  opinions  of  the  CTS  training 
exereises  in  terms  of  level  of  ehallenge  and  value  of  the  feedbaek  (mean  ratings  of  4.7  and  4.9, 
respeetively).  These  are  seant  data  but  suggestive  of  a  positive  regard  on  the  part  of  Soldiers  who 
had  developed  a  working  familiarity  with  the  CTS  eapabilities  (reeall  from  the  usage  data  in 
Table  14  that  more  partieipants  worked  with  the  pre-  and  post-test  eomponents  of  the  CTS 
modules  than  the  aetual  training  lessons).  Here  again  the  rating  framework  was  global,  shedding 
no  light  on  the  individual  CTS  modules. 

Table  22 

User  Jury  Mean  Ratings  for  Training  Aspects  of  LASTS,  by  Module 


Training  Aspect 

Module* 

Grand 

Mean 

ESA 

n=15 

SSA 

n=6 

SR 

n=9 

AE 

n=7 

The  module  appears  easy  to  use 

3.6 

3.2 

4.7 

4.9 

4.1 

The  module  relates  directly  to  me  and  my  career 

2.5 

1.2 

3.2 

3.6 

2.6 

The  module  covers  important  aspects  of  leader  development 

3.4 

3.7 

3.8 

3.6 

3.6 

The  module  makes  sense  based  on  my  experience 

3.4 

1.8 

4.1 

3.4 

3.2 

The  exercises  focus  on  realistic  situations  and  challenges 

2.7 

1.3 

2.9 

2.0 

2.2 

The  training  exercises  are  challenging 

2.2 

1.3 

1.6 

2.2 

1.8 

The  feedback  from  the  exercises  is  a  valuable  teaching  tool 

3.1 

1.0 

2.4 

2.6 

2.3 

Grand  Mean 

3.0 

1.9 

3.2 

3.2 

*  ESA  =  Effective  Self-Appraisal;  SSA  =  Self  and  Social  Awareness;  SR  =  Self  Regulation;  AE  = 
Application  Exercise. 

Note'.  The  rating  scale  ranged  from  1  (strongly  disagree)  to  6  (strongly  agree),  with  no  neutral 
point. 


The  user  juries  generated  eonsiderably  more  data  on  the  training  aspeets  of  the  CTS 
modules  (Table  23).  The  table’s  summary  of  the  relevant  rating  results  eovers  all  eight  modules. 
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each  examined  by  a  separate  group  of  Soldiers.  While  the  opinions  of  training  aspects — 
informativeness,  responsiveness,  exercise  challenge,  and  feedback  value — showed  a  weakly 
favorable  pattern  on  the  whole,  the  ratings  varied  across  the  different  modules.  Module  #8 
(Decide  when  to  seek  information)  garnered  relatively  strong  positive  regard  (grand  mean,  5.4), 
while  four  modules  (#3,  #5,  #6,  #7)  drew  weakly  favorable  opinions  overall  (grand  means  of  4.4 
to  4.8).  The  remaining  three  modules  (#1,  #2,  #4)  received  overall  ratings  that  were  slightly 
unfavorable  (grand  means  of  3.4  to  3.7).  These  findings  may  indicate  that  the  perceived  quality 
differed  between  the  various  CTS  modules.  However,  each  user  jury  rated  only  two  of  the  eight 
CTS  modules  and  the  user  juries  varied  along  several  dimensions  (installation,  facilitator, 
background,  etc.),  so  the  differences  between  user  juries  may  have  contributed  to  the  patterns 
seen  in  Table  23. 

The  unit  participants’  ratings  of  four  different  training  aspects  of  the  AXL  vignettes  are 
included  in  Table  21.  These  results  come  from  the  user  satisfaction  survey,  which  did  not  ask  the 
participants  to  distinguish  between  the  Power  Hungry  and  Tripwire  vignettes.  The  mean  ratings 
indicated  moderately  favorable  opinions  regarding  the  training  dimensions  built  into  the  films  as 
well  as  the  group-centered  procedures.  The  means  were  5.0  or  greater  with  68-80%  of  the 
respondents  agreeing  (slightly,  somewhat,  strongly)  with  any  given  questionnaire  statement.  The 
viewpoint  diversity  dimension  associated  with  group  discussion  drew  especially  favorable 
responses  (M=  5.7),  consistent  with  participants’  positive  comments  about  the  benefits  of  group 
interaction.  The  consistently  positive  ratings  for  the  AXL  vignettes  provide  persuasive  evidence 
of  favorable  reception  among  the  target  audience. 

Table  23 

User  Jury  Mean  Ratings  for  Training  Aspects  of  CTS  Modules 


CTS  Module 

* 

1 

2 

3 

4 

5 

6 

7 

8 

Training  Aspect 

n=12 

17=6 

17=10 

17=8 

17=12 

17=11 

17=26 

17=17 

The  training  module  was  informative 

3.5 

4.0 

4.7 

3.8 

4.8 

5.4 

5.3 

5.6 

The  training  tool  was  responsive  to  my  educational  needs 

3.0 

3.0 

4.1 

3.1 

4.3 

4.4 

4.2 

5.1 

The  training  exercises  were  challenging 

3.4 

3.7 

5.0 

3.1 

4.1 

4.5 

4.0 

5.5 

The  feedback  from  exercises  was  a  valuable  teaching  tool 

3.8 

4.2 

3.7 

4.0 

4.9 

4.7 

5.1 

5.2 

Grand  Mean 

3.4 

3.7 

4.4 

3.5 

4.5 

4.8 

4.6 

5.4 

*  The  rating  scale  ranged  from  1  (strongly  disagree)  to  7  (strongly  agree),  with  4  being  neutral. 


Many  of  the  user  jury  members  who  worked  with  the  tools  in  some  detail  stated  that 
LASTS  and  CTS  (no  user  juries  reviewed  AXL)  required  major  revisions  to  be  effective. 
Recurring  themes  were  voiced  as  essential  for  user  acceptability: 

•  The  materials  must  be  clearly  designed  for  Army  officers  or  NCOs  and  not  perceived 
as  adaptations  of  materials  designed  for  an  academic  or  a  civilian  audience. 

•  Scenarios  or  anecdotes  must  be  relevant  to  Army  operations,  current,  and  realistic. 
Some  viewed  these  elements  as  trivial  or  overly  simplistic. 
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•  The  instructional  materials  must  reflect  understanding  of  the  operational  Army.  The 
participants  sometimes  criticized  the  instructional  presentation  as  patronizing  and 
amateurish.  Specific  criticisms  included; 

o  Use  of  academic  language  and  tone  (defeats  credibility  and  acceptability), 
o  Use  of  unfamiliar  terminology  (degrades  Soldier  friendliness), 
o  Presence  of  unnecessary  or  redundant  information  (detracts  seriously). 

•  All  instructional  materials  must  be  clear  and  concise  (to  the  point,  easy  to  read,  easy  to 
follow).  Otherwise,  the  learner  will  tune  out  quickly. 

•  The  instruction  should  make  the  learner’s  job  as  easy  as  possible.  On  this  count  the 
following  rules  emerged: 

o  Don’t  expect  the  learner  to  read  run-on  text. 

o  Break  run-on  text  into  chunks  of  information  such  as  concise  bullets, 
o  Include  graphic  methods  such  as  charts  to  present  knowledge, 
o  Simplify  wherever  possible  to  enhance  conceptual  clarity, 
o  Put  the  bottom  line  up  front. 

o  Use  more  than  one  page  where  information  is  crowded. 
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Learning  Effectiveness  of  the  Tools 


LASTS  Modules 

Among  the  user  satisfaction  survey  queries  were  items  addressing  the  training  outcomes 
of  the  ALD  training  tools.  For  the  LASTS  tool,  Table  24  summarizes  the  unit  participants’ 
ratings  of  two  outcome  aspects.  The  survey  given  to  the  unit  participants  did  not  distinguish 
between  the  LASTS  modules,  so  the  ratings  apply  at  the  global  level.  The  results  in  the  table 
indicate  moderately  favorable  opinions  about  the  information  gained  from  the  exercises  (mean 
rating,  4.8)  and  the  resulting  attention  focused  on  self-appraisal  (mean  rating,  4.8).  Although  they 
represent  only  two  outcome  aspects,  the  results  suggest  that  Soldiers  formed  a  positive  regard  for 
the  LASTS  capabilities,  based  on  their  online  work  with  the  tool’s  modules. 

Table  24 

Unit  Participants’  Ratings  of  Training  Outcomes  of  LASTS  Modules  (N  =  33) 


Training  Outcome 

Mean 

Rating* 

% 

Agreeing 

%  Dis¬ 
agreeing 

The  exercises  provided  useful  information  and  techniques 

4.8 

66% 

3% 

The  training  taught  me  to  think  about  self-appraisal 

4.8 

59% 

6% 

*  The  rating  scale  ranged  from  1  (strongly  disagree)  to  7  (strongly  agree),  with  4  being  neutral. 


For  the  user  juries.  Table  25  presents  the  mean  ratings  for  six  outcome  aspects  of  four 
LASTS  modules.  In  contrast  with  the  results  from  the  unit  participants,  the  user  jury  ratings  tend 
to  reflect  slightly  to  moderately  negative  opinions  about  the  LASTS  modules.  (Because  the  user 
jury  rating  scale  had  no  neutral  point,  a  “neutraf’  average  would  be  3.5).  Five  outcome  aspects 
— utility  of  exercise  information/techniques,  personal  insights,  etc. — drew  mildly  unfavorable 
ratings,  with  grand  means  from  2.6  to  2.8.  Notably,  the  Self  and  Social  Awareness  module 
received  an  overall  rating  (grand  mean,  2.2)  that  was  noticeably  lower  than  the  grand  means  for 
the  other  three  modules.  This  is  consistent  with  the  user  jury  results  for  training  aspects  of  the 
same  module  (see  Table  22).  It  should  be  remembered  that  the  user  jury  members  had  no  post¬ 
training  application  experience  with  which  to  judge  the  utility  of  the  ALD  tools. 

CTS  Modules 

With  an  eye  on  gauging  the  impact  of  CTS  training  on  critical  thinking  abilities,  the 
online  pre-  and  post-test  for  each  module  were  intended  to  measure  performance  before  and  after 
completing  the  lessons.  Presumably  the  learning  effects  of  the  lessons  would  increase  the  score 
on  the  post-test  compared  to  the  pre-test.  Each  test  contained  a  unique  combination  of  questions 
that  varied  in  number,  with  a  pre-defined  scoring  template.  The  recorded  performance  of  unit 
participants  on  the  CTS  pre-  and  post-tests  is  summarized  in  Table  26.  By  virtue  of  self¬ 
selection,  the  subset  of  unit  participants  completing  any  given  test  varied  across  modules,  with 
variable  overlap  between  the  pre-  and  post-test  sub-samples.  Because  of  the  erratic  pattern  with 
which  participants  completed  pre-tests,  training  lessons,  and  post-tests  (discussed  in  the  next 
paragraph),  the  data  are  suggestive  at  best. 
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Table  25 

User  Jury  Mean  Ratings  for  Training  Outcomes  of  LASTS  Modules 


Training  Outcome 

Module* 

Grand 

Mean 

ESA 

n=15 

SSA 

n=6 

SR 

n=9 

AE 

n=7 

The  module  provides  useful  information 

3.5 

3.7 

3.1 

4.3 

3.6 

The  module  gave  me  insights  on  my  self-development  approach 

2.8 

2.3 

2.9 

3.4 

2.8 

The  module  added  to  my  understanding  of  self-development 

2.8 

2.0 

2.3 

3.1 

2.6 

The  exercises  provide  useful  information  and  techniques 

3.1 

2.0 

2.2 

3.3 

2.6 

The  training  taught  me  how  to  improve  my  self-development 

2.5 

1.7 

3.0 

3.4 

2.6 

The  module  taught  me  to  think  about  my  self-development 

3.1 

1.8 

2.5 

2.9 

2.6 

Grand  Mean 

3.0 

2.2 

2.7 

3.4 

*  ESA  =  Effective  Self-Appraisal;  SSA  =  Self  and  Social  Awareness;  SR  =  Self  Regulation;  AE  = 
Application  Exercise. 

Note'.  The  rating  scale  ranged  from  1  (strongly  disagree)  to  6  (strongly  agree),  with  no  neutral 
point. 


Table  26 

Unit  Participants  ’  Scores  (Percent  Correct)  for  CTS  Pre-  and  Post-Tests,  by  Module 


Module 

Pre-Test 

Post-Test 

Difference 

n 

Mean 

Range 

n 

Mean 

Range 

Delta 

1 

16 

52.5 

6-94 

22 

56.1 

35-72 

+3.6 

2 

13 

47.2 

24-71 

20 

56.4 

25-94 

+9.3 

3 

13 

49.4 

20-91 

18 

47.5 

5-70 

-1.9 

4 

11 

N/A® 

N/A 

18 

56.8 

32-91 

N/A 

5 

1 

WA^ 

(65) 

5 

43.0 

30-55 

N/A 

6 

1 

WA^ 

(32) 

5 

35.0 

25-60 

N/A 

7 

1 

WA^ 

(58) 

5 

46.2 

27-62 

N/A 

8 

0 

N/A 

N/A 

5 

34.0 

25-50 

N/A 

®  Data  were  missing. 

The  sub-sample  size  (n  =  1)  did  not  support  computing  a  mean. 


Frequently  partieipants  who  did  take  pre-  and/or  post-tests  did  not  aeeess  the  training 
lessons,  and  those  who  did  aeeess  them  normally  spent  too  little  time  to  learn  the  material. 
Several  examples  of  individual  use  of  the  CTS  modules  illustrate  the  issues  with  the  performanee 
data  from  the  pre-  and  post- tests. 

•  One  partieipant  took  both  the  pre-  and  post  tests  for  the  first  3  modules,  with  no 
training  aeeessed.  His  seores  went  down  12  points  for  Module  1,  up  7  points  for 
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Module  2,  and  up  by  33  points  for  module  3.  In  the  ease  of  Module  4  he  took  only  the 
post-test  without  aeoessing  the  other  eomponents. 

•  Another  partieipant  took  all  eight  post-tests,  without  aeoessing  any  pre-tests  or  training 
eomponents.  His  times  for  the  first  two  modules  were  25.2  and  36.7  min  (soores  of  50 
and  58,  respeotively).  Times  then  deoreased  to  under  20  min  for  Modules  3  and  4 
(soores  of  45  and  60,  respeotively),  and  then  to  10.5  min  for  module  5  (soore  =  55).  He 
aooessed  the  three  remaining  modules  for  less  than  10  min  eaoh,  posting  soores  of  40, 
50  and  40,  respeotively. 

•  A  partieipant  spent  less  than  3  min  on  the  pre-test  for  Module  1  and  then  spent  17  min 
taking  the  post-test  (no  training  time  available)  to  have  his  soore  deorease  from  94  to 
64.  Similarly,  this  partieipant  took  both  tests  for  Module  3  (13.0  min  on  the  pre-test 
and  33.0  min  on  the  post-test)  with  his  soore  deolining  from  91  to  55. 

•  Another  partieipant  took  the  Module  1  pre-test  in  8.2  min  and  spent  188.9  min  (3.1  hr) 
on  the  lessons,  after  whioh  his  post-test  soore  (70)  exoeeded  his  pre-test  soore  by  9 
points.  On  Module  2,  with  only  24.8  min  of  training,  his  soores  went  from  55  points  on 
the  pre-test  to  94  points  on  the  post-test.  He  also  aooessed  Module  3  for  99.9  min  but 
took  only  the  pre-test. 

•  One  partieipant  took  no  pre-tests  and  aooessed  no  lessons.  He  aooessed  the  post-tests 
for  Modules  2-8,  spending  less  than  2  min  on  eaoh  exoept  for  4.3  min  on  post-test  #4. 

Providing  an  alternative  look  at  the  CTS  performanoe  tests  were  the  user  juries,  whose 
members  oompleted  the  pre-  and  post-tests  in  hardoopy  form.  Table  27  summarizes  the  user  jury 
results  by  module,  inoluding  the  pre-to-post  “deltas.”  Not  surprisingly,  the  average  performanoe 
shows  wide  variability  aoross  modules,  with  pre-test  means  ranging  from  25%  oorreot  to  81%, 
and  post-test  means  ranging  from  46%  to  68%.  The  deltas  indioate  performanoe  improvement  for 
some  CTS  modules  and  performanoe  deoline  for  others. 

Table  27 

User  Jury  Scores  (Percent  Correct)  for  CTS  Pre-  and  Post-Tests,  by  Module 


Module 

Pre-Test 

Post-Test 

Difference 

n 

Mean 

Range 

n 

Mean 

Range 

Delta 

1 

12 

72.2 

50-100 

12 

58.9 

47-72 

-13.3 

2 

6 

69.0 

36-86 

6 

56.0 

22-69 

-13.0 

5 

12 

50 

27-73 

12 

57.8 

37-68 

+7.8 

6 

11 

40.5 

27-59 

11 

52.5 

30-70 

+  12.0 

7 

27 

80.9 

36-91 

26 

68.5 

39-93 

-12.4 

8 

18 

25.4 

8-50 

17 

46.2 

25-65 

+20.8 

Note:  Data  for  modules  3  and  4  are  omitted  because  they  were  compiled  improperly. 


Interpreting  the  results  in  Table  27  is  diffioult  beoause  of  oonfounding  faotors.  Most 
notably,  the  user  jury  Soldiers  did  not  engage  in  self-paoed  training  at  individual  workstations, 
but  viewed  the  faoilitator-oontrolled  display  on  a  large  projeotion  soreen  along  with  the  rest  of 
the  group.  Further,  they  partioipated  freely  in  open  disoussions  during  the  training  session,  whioh 
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may  have  distracted  them  by  competing  for  their  attention  and  exposing  them  to  others’ 
reactions.  Finally,  taking  the  tests  in  hardcopy  form  may  have  distorted  the  performance 
dynamics  intended  by  the  CTS  developers.  As  a  result,  the  performance  outcomes  seen  in  Table 
27  cannot  be  taken  as  representative  of  proficiency  levels  or  potential  improvements  attributable 
to  online  CTS  training. 

The  user  satisfaction  survey  asked  the  unit  participants  to  rate  a  few  training  outcome 
aspects  of  the  CTS  tool,  with  no  distinction  between  specific  modules,  using  the  familiar 
agree/disagree  scale.  The  rating  results  (Table  28)  indicate  moderately  favorable  opinions  on  the 
how-to  contributions  of  the  training.  The  means  ranged  from  4.7  to  4.9,  with  half  to  two-thirds  of 
the  sample  agreeing  (slightly,  somewhat,  strongly)  with  the  questionnaire  statements  about 
training  outcomes.  These  results  for  the  CTS  outcomes  provide  additional  evidence  of  favorable 
reception  of  this  tool  among  the  target  audience. 

Table  28 

Unit  Participants  ’  Ratings  of  Training  Outcomes  of  CTS  Modules  (N  =  33) 


Training  Outcome 

Mean 

Rating* 

% 

Agreeing 

%  Dis¬ 
agreeing 

The  training  taught  me  how  to  listen,  analyze,  and  evaluate  messages 

4.9 

67% 

12% 

The  training  taught  me  to  think  about  the  big  picture 

4.8 

63% 

9% 

The  training  helped  me  perform  a  self-appraisal  of  my  leadership  skills 

4.7 

53% 

13% 

*  The  rating  scale  ranged  from  1  (strongly  disagree)  to  7  (strongly  agree),  with  4  being  neutral. 


Parallel  data  on  training  outcomes  of  the  CTS  modules  came  from  user  juries  via  the 
CTS-specific  user  satisfaction  survey.  As  Table  29  shows,  the  mean  ratings  in  the  majority  of 
cases  were  moderately  favorable,  reaching  as  high  as  5.8.  However,  the  ratings  varied 
considerably  across  the  different  modules.  Five  modules  (#3,  #5,  #6,  #7,  #8)  drew  moderately 
favorable  opinions  overall  (grand  means  of  4.6  to  5.3).  The  remaining  three  modules  (#1,  #2,  #4) 
received  overall  ratings  that  were  mildly  unfavorable  (grand  means  of  3.3  to  3.7).  These  findings 
are  very  similar  to  and  support  the  validity  of  the  patterns  found  in  Table  23  (CTS  training 
aspects),  where  modules  #1,  #2  and  #4  also  received  the  lowest  ratings  of  the  CTS  set.  These 
results  suggest  that  the  perceived  utility  differed  between  the  various  CTS  modules.  However, 
the  user  juries  varied  along  several  dimensions  (installation,  facilitator,  background,  etc.)  and  the 
differences  between  groups  may  have  contributed  to  the  patterns  seen  in  Table  29. 

AXL  Vignettes 

The  interactive  courseware  for  the  two  AXL  vignettes  challenged  the  unit  participants  to 
rank  order  the  top  seven  issues  from  a  list  of  21  actions  that  Soldiers  in  the  film  vignette  should 
have  done  differently.  The  assigned  ranks  were  then  used  to  compute  scores  indicating  how 
much  emphasis  a  participant  placed  on  four  different  categories  of  issues.  If  obtained  before  and 
after  AXL  training,  the  category  scores  could  reveal  shifts  in  participant  thinking  due  to  the 
vignette -based  learning.  Table  30  summarizes  the  end-of-session  emphasis  scores  for  the  two 
film-based  vignettes,  comparing  the  cultural  awareness  module  with  the  leadership  values 
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module.  The  means  varied  substantially  between  the  four  eategories — by  a  faetor  of  4  in  the  ease 
of  the  eultural  awareness  module  for  the  Tripwire  vignette.  The  patterns  were  similar  for  the  two 
vignettes,  exeept  for  the  eategory  yielding  the  lowest  mean  seores  (self-regulation  issues  for 
Power  Hungry,  and  eultural  issues  for  Tripwire).  However,  the  results  in  Table  30  shed  no  light 
on  training  effeetiveness  beeause  there  was  no  pre-versus-post  eomparison. 

Table  29 

User  Jury  Mean  Ratings  of  Training  Outcomes  of  CTS  Modules 


CTS  Module* 

1 

2 

3 

4 

5 

6 

7 

8 

Training  Outcome 

n=12 

17=6 

n=10 

n=8 

n=12 

n=11 

n=26 

n=17 

The  training  taught  me  to  listen,  analyze,  and  evaluate  msgs 

3.4 

3.8 

4.3 

3.0 

4.5 

4.7 

4.3 

5.1 

The  training  taught  me  to  think  about  the  big  picture 

4.2 

4.2 

4.9 

3.9 

5.0 

5.1 

4.8 

5.4 

Questions  made  me  think  critically  about  scenario  issues 

3.9 

4.0 

5.5 

3.8 

5.7 

5.4 

5.4 

5.8 

The  training  helped  me  self-appraise  my  leadership  skills 

3.2 

2.7 

3.5 

2.4 

3.5 

4.5 

4.5 

4.9 

Grand  Mean 

3.7 

3.7 

4.6 

3.3 

4.7 

4.9 

4.8 

5.3 

*  The  rating  scale  ranged  from  1  (strongly  disagree)  to  7  (strongly  agree),  with  4  being  neutral. 
Table  30 

Unit  Participants  ’  Scores  on  AXL  Relative  Emphasis  Scales,  by  Vignette  and  Module 


Category 

Cultural  Awareness  Module 

Leadership  Values  Module 

Mean 

Std  Dev 

Range 

Mean 

Std  Dev 

Range 

Power  Hungry  Vignette 

CNI 

00 

II 

<: 

CD 

1^ 

II 

<: 

Tactical  Issues  Emphasis 

7.40 

5.83 

0-18 

6.49 

6.17 

0-18 

Leader-Subordinate  Emphasis 

6.37 

5.23 

0-22 

6.14 

6.96 

0-25 

Cultural  Issues  Emphasis 

5.82 

5.00 

0-17 

3.63 

4.60 

0-15 

Self  Regulation  Emphasis 

2.95 

3.89 

0-15 

2.16 

3.49 

0-15 

Tripwire  Vignette 

II 

N=^5 

Tactical  Issues  Emphasis 

10.32 

8.63 

0-27 

9.40 

8.58 

0-28 

Leader-Subordinate  Emphasis 

6.32 

6.15 

0-18 

5.53 

5.25 

0-18 

Cultural  Issues  Emphasis 

2.55 

3.76 

0-15 

2.53 

3.14 

0-8 

Self  Regulation  Emphasis 

4.02 

4.73 

0-14 

6.47 

6.97 

0-18 

Note:  The  scores  were  derived  from  relative  importance  rankings  of  21  issues. 


Impact  and  Value  of  Using  the  Tools 

The  user  satisfaetion  survey  ineluded  items  addressing  seleeted  impaet  aspeets  of  the 
ALD  training — partieularly  usefulness  and  benefits  of  using  the  various  tools.  Table  31 
summarizes  the  agree/disagree  ratings  provided  by  the  unit  partieipants  at  the  end  of  their  unit’s 
operational  demonstration  period.  Three  of  the  diagnostie  tools  (MLQ,  TKML,  and  personality 
inventory)  drew  moderately  favorable  opinions  (means  above  4.5)  on  their  usefulness.  The 
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personality  inventory  received  the  most  positive  ratings  of  usefulness  {M=  5.0),  perhaps 
reflecting  its  broad  applicability  and  detailed  feedback.  The  results  suggest  that  the  participants 
perceived  practical  value  in  the  three  selected  diagnostic  instruments.  The  survey  did  not  include 
comparable  items  for  the  other  diagnostic  tools. 

Table  31 

Unit  Participants  ’  Ratings  of  Selected  Impact  Aspects  of ALD  Training 


Impact  Aspect 

N 

Mean 

Rating* 

% 

Agreeing 

%  Dis¬ 
agreeing 

Tool-Specific  Dimensions 

The  MLQ  was  a  useful  tool 

34 

4.6 

59% 

9% 

The  TKML  feedback  from  SMEs  was  useful 

34 

4.6 

68% 

15% 

The  personality  Inventory  was  a  useful  tool 

34 

5.0 

74% 

9% 

The  LASTS  exercises  provided  useful  Information  &  techniques 

32 

4.8 

66% 

3% 

The  LASTS  modules  provided  definite  benefits 

32 

4.9 

59% 

6% 

The  Information  gained  from  CTS  exercises  was  useful 

33 

5.2 

73% 

3% 

The  CTS  training  was  easily  Implemented  Into  the  real  world 

33 

4.6 

58% 

15% 

The  feedback  from  the  AXL  actors  provided  leadership  Insights 

40 

5.1 

70% 

10% 

Overall  Program  Dimensions 

1  am  a  better  leader  after  completing  the  ALD  training 

39 

4.5 

46% 

15% 

The  ALD  training  changed  my  leadership  style 

39 

3.5 

26% 

36% 

The  benefits  outweighed  the  time  required  for  ALD  training 

38 

4.2 

47% 

26% 

The  ALD  training  was  easily  Implemented  Into  the  real  world 

39 

4.7 

59% 

13% 

1  recommend  ALD  training  for  all  Individuals  In  leadership  positions 

39 

4.7 

54% 

10% 

The  ALD  training  Interfered  with  my  everyday  training  mission 

37 

5.0 

65% 

16% 

*  The  rating  scale  ranged  from  1  (strongly  disagree)  to  7  (strongly  agree),  with  4  being  neutral. 


As  seen  in  Table  31,  the  three  training  tools  (LASTS,  CTS,  and  AXL  modules)  received 
moderately  favorable  ratings  of  their  usefulness,  benefits,  and  applicability.  The  mean  ratings 
ranged  from  4.6  to  5.2,  with  58-73%  of  the  sample  agreeing  (slightly,  somewhat,  strongly)  with 
the  survey’s  positive  statements.  The  CTS  and  AXL  tools  drew  the  strongest  positive  ratings,  but 
it  would  be  unwarranted  to  infer  meaningful  differences  between  the  three  ALD  training 
products.  Altogether,  the  results  indicate  that  the  participants  saw  potential  value  in  using  the 
toolkit’s  training  capabilities. 

The  unit  participants  also  rated  several  dimensions  of  the  overall  ALD  program.  As  Table 
31  shows,  they  agreed  to  a  mildly  positive  degree  (M=  4.5  to  4.7)  that  the  ALD  training  (a) 
made  them  better  leaders,  (b)  could  be  easily  implemented  in  the  real  world,  and  (c)  could  be 
recommended  for  leaders  in  general.  At  the  same  time,  the  respondents  were  ambivalent  (M  = 
4.2)  about  the  ALD  benefits  outweighing  the  required  training  time.  Not  surprisingly,  the  sample 
mildly  disagreed  (M=  3.5)  that  ALD  training  changed  their  leadership  style.  And  they  felt 
moderately  strongly  (M=  5.0)  that  the  ALD  training  interfered  with  their  routine  training 
mission,  perhaps  refiecting  the  pressures  of  the  COE. 
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Additional  data  on  the  impact  and  value  of  the  ALD  toolkit  came  from  the  user  juries. 

The  user  jury  results  for  the  LASTS  modules  (Table  32)  consistently  convey  ambivalent  (grand 
means  of  3.2  or  3.3)  or  moderately  negative  (grand  means  of  2.0  or  2.1)  opinions  about  practical 
aspects  of  the  LASTS  modules  examined.  Across  all  four  user  juries,  the  members  disagreed 
moderately  (mean  ratings  from  1.2  to  2.3)  that  all  five  modules  are  worth  completing.  Consistent 
with  results  presented  above,  the  Self  and  Social  Awareness  module  drew  less  favorable  opinions 
(grand  mean,  2.1)  than  the  other  three  modules  (M=  3.3  in  each  case).  On  the  whole  these  results 
suggest  that  the  LASTS  modules  lack  the  practical  value  that  Army  leaders  expect  for  self¬ 
development  purposes.  However,  the  user  jury  results  differ  from  the  slightly  more  positive 
opinions  of  the  unit  participants  (e.g..  Table  24),  so  interpretive  caution  should  be  exercised. 

Table  32 

User  Jury  Mean  Ratings  of  Practical  Aspects  of  LASTS  Modules 


Practical  Aspect 

Module* 

Grand 

Mean 

ESA 

n=15 

SSA 

n=6 

SR 

n=9 

AE 

n=7 

I’d  work  thru  all  five  modules  because  1  think  they’re  valuable 

2.3 

1.2 

2.2 

2.3 

2.0 

1  can  see  some  definite  benefits  of  the  LASTS  modules 

3.5 

2.2 

3.9 

3.7 

3.3 

1  can  see  how  1  could  apply  what’s  taught  to  my  career 

3.5 

2.5 

3.4 

3.5 

3.2 

1  think  LASTS  training  could  make  me  a  better  leader 

3.4 

2.5 

3.6 

3.7 

3.3 

1  think  LASTS  training  would  be  good  for  leaders  1  know 

3.7 

2.2 

3.3 

3.5 

3.2 

Grand  Mean 

3.3 

2.1 

3.3 

3.3 

*  ESA  =  Effective  Self-Appraisal;  SSA  =  Self  and  Social  Awareness;  SR  =  Self  Regulation;  AE  = 
Application  Exercise. 

Note:  The  rating  scale  ranged  from  1  (strongly  disagree)  to  6  (strongly  agree),  with  no  neutral 
point. 


Table  33  summarizes  the  user  jury  ratings  of  practical  aspects  of  the  CTS  modules.  Not 
surprisingly,  the  ratings  varied  considerably  across  the  different  modules.  Five  modules  (#3,  #5, 
#6,  #7,  #8)  drew  moderately  favorable  opinions  overall  (grand  means  of  4.5  to  5.2).  The 
remaining  three  modules  (#1,  #2,  #4)  received  mildly  negative  ratings  overall  (grand  means  of 
3.3  to  3.4).  These  findings  are  very  similar  to  the  patterns  in  Table  23  (CTS  training  aspects)  and 
Table  29  (CTS  training  outcomes),  indicating  consistent  opinions  across  a  variety  of  dimensions. 
It  should  be  remembered  that  the  user  juries  differed  in  several  respects  (facilitator,  background, 
etc.)  and  the  patterns  seen  in  Table  33  may  reflect  the  basic  differences  between  groups. 

Some  of  the  items  in  the  user  satisfaction  survey  given  to  the  LASTS  user  jury  members 
addressed  general  context  issues  that  may  speak  to  the  potential  value  of  the  ALD  toolkit.  The 
pertinent  items  appear  in  Table  34  along  with  a  summary  of  agree/disagree  ratings.  The  sample 
of  Soldiers  agreed  moderately  (M=  5.1  on  the  6-point  scale)  that  leader  development  planning  is 
good  for  an  Army  officer’s  career.  Regarding  their  satisfaction  with  today’s  leader  development 
realities,  the  Soldiers  registered  ambivalent  opinions  (mean  ratings  of  3.4  to  3.6).  Perhaps  more 
important,  CTS  user  jury  members  (not  represented  in  Table  34)  agreed  with  moderate  strength 
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(M=  6.1  on  a  7-point  scale,  N=  99)  that  creating  a  leadership  plan  and  goals  is  necessary  to  be 
an  effective  leader,  as  did  unit  participants  {M=  5.9,  N=  33).  These  results  point  to  the  need  for 
leader  development  planning,  but  the  survey  did  not  ask  the  participants  what  planning  tools 
might  be  useful. 


Table  33 

User  Jury  Mean  Ratings  of  Practical  Aspects  of  CTS  Modules 


CTS  Module* 

1 

2 

3 

4 

5 

6 

7 

8 

Practical  Aspect 

n=12 

rpQ 

n=10 

n=8 

n=12 

n=11 

n=26 

n=17 

The  training  package  is  a  valuable  leader  development  tool 

3.2 

3.3 

4.4 

3.6 

5.3 

4.9 

4.9 

6.2 

The  information  gained  from  the  exercises  was  useful 

3.2 

3.5 

4.5 

3.6 

4.7 

4.6 

4.9 

5.4 

Training  will  be  easily  implemented  into  real-world  application 

4.1 

3.8 

4.3 

4.1 

5.2 

5.1 

5.0 

5.1 

Benefits  outweigh  the  time  required  to  complete  the  training 

3.3 

3.0 

5.0 

2.2 

5.4 

5.0 

4.9 

4.5 

1  recommend  this  training  for  all  leaders 

3.2 

3.3 

4.1 

2.8 

5.0 

5.0 

4.7 

4.9 

Grand  Mean 

3.4 

3.4 

4.5 

3.3 

5.1 

4.9 

4.9 

5.2 

*  The  rating  scale  ranged  from  1  (strongly  disagree)  to  7  (strongly  agree),  with  4  being  neutral. 
Table  34 

User  Jury  (LASTS)  Mean  Ratings  for  Army  Leader  Development  Aspects 


Statement 

N 

Mean 

Rating* 

% 

Agreeing 

Leader  development  planning  is  good  for  an  Army  officer’s  career 

37 

5.1 

97% 

1  am  satisfied  with  the  Army’s  leader  development  tools  and  practices 

32 

3.6 

56% 

1  am  satisfied  with  my  unifs  leader  development  priorities  and  resources 

29 

3.5 

48% 

Leader  development  competes  with  my  everyday  training  mission 

37 

3.4 

51% 

*  The  rating  scale  ranged  from  1  (strongly  disagree)  to  6  (strongly  agree),  with  no  neutral  point. 


Lessons  Learned 

Throughout  the  course  of  the  demonstration  and  assessment,  the  research  team  gathered 
numerous  lessons  about  the  ALD  program.  Presented  in  this  section  are  major  lessons  that  hold 
value  for  future  program  architects,  training  developers,  and  investigators.  They  highlight 
substantive  considerations  regarding  leader  self-development  and  delivery  of  online  training.  In 
many  cases,  the  lessons  point  to  a  need  for  further  exploration.  The  following  sub-sections 
organize  the  discussion: 

•  The  Big  Picture 

•  Program  of  Instruction 

•  Learning  Conditions 

•  Fielding  and  Implementation 
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The  Big  Picture 


Army  Policy  versus  Unit  Schedules 

The  Army’s  published  policy  for  leadership  development  (U.S.  Department  of  the  Army, 
2007)  specifies  self-development  as  one  of  three  domains  involved  in  developing  competent 
leaders.  It  assigns  all  leaders  the  responsibility  for  developing  those  junior  to  them  through 
counseling,  coaching,  and  mentoring.  However,  in  tactical  units  the  policy  comes  into  conflict 
with  the  persistently  high  OPTEMPO  and  the  overcrowded  docket  of  deployment-driven 
requirements  that  extend  through  all  phases  of  the  unit  lifecycle.  The  conflict  is  resolved  by 
relegating  the  self-development  requirements  to  a  status  outside  the  unit  training  schedule,  which 
forces  unit  leaders  to  create  time  for  their  own  and  others’  self-development,  mentoring  and 
coaching  activities  by  unofficial  means.  In  reality,  this  leaves  little  to  no  time  available  for  self¬ 
development  in  units. 

Interdependence  among  Programs 

As  a  large  organization,  the  Army  has  numerous  programs  directed  at  developing  and 
sustaining  competent  leaders.  Most  of  them  claim  an  official  status,  while  some  represent  grass¬ 
roots  initiatives  (e.g.,  communities  of  practice  such  as  CompanyCommand.mil).  When  new 
leader  development  technologies  or  tools  become  available,  they  join  the  existing  constellation 
of  programs.  Whether  specified  or  implied,  a  new  program’s  relationships  with  existing 
programs  can  substantially  influence  how  the  target  audience  perceives  the  new  capabilities.  In 
early  program  stages,  a  system-of-systems  framework  could  facilitate  design  and  development  of 
new  leader  development  tools.  The  same  framework  could  enhance  efforts  to  evaluate  and 
implement  the  tools.  The  program  proponent  and  developers  should  invest  effort  in  making  it 
clear  to  the  target  audience  how  the  new  tools  fit  into  the  total  Army  picture.  This  can  help  orient 
the  intended  users,  expand  their  own  knowledge  of  programs,  and  establish  a  measure  of 
credibility  for  the  new  program. 

Distributed  Learning  Trend 

The  Army’s  growing  reliance  on  distributed  learning  extends  to  the  leader  self¬ 
development  arena.  Anywhere-anytime  training  is  touted  as  a  solution  to  the  shortage  of  time 
available  for  self-development  tasks.  But  the  skepticism  expressed  by  many  demonstration 
participants  raises  questions  about  leveraging  the  attractive  features  of  online  delivery  methods. 
To  achieve  its  distributed  learning  goals  in  the  leader  development  arena,  the  Army  faces  several 
challenges.  First,  online  programs  that  give  credit  for  completion  without  study  and  effort  are  not 
taken  seriously.  Soldiers  must  be  able  to  see  how  their  investment  in  legitimate  learning  brings 
tangible  value  in  terms  of  career  advancement.  Second,  the  training  audience  may  not  accept  the 
appropriateness  of  using  online  tools  without  being  convinced.  Army  leaders  favor  face-to-face 
methods  for  developing  their  leadership  skills,  largely  because  they  highly  value  interpersonal 
interaction.  Third,  tactical  units  do  not  necessarily  possess  the  computer  resources  and 
infrastructure  (network  capabilities)  needed  to  empower  the  intended  users.  Fourth,  online  tools 
suffer  a  distinct  disadvantage  if  they  do  not  compare  favorably  with  the  wide  assortment  of  Web 
sites  and  games  available  to  Soldiers  beyond  the  boundaries  of  their  jobs. 
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Current  Reality  of  Leadership  Development 

Discussions  with  Soldiers  indicate  that  leadership  development,  other  than  that  which 
occurs  as  a  product  of  training  exercises  or  routine  duties,  is  not  occurring  formally  to  any  major 
degree  in  operational  units.  Leader  development  programs  do  exist  in  units  (Schirmer,  Crowley, 
et  al,  2008),  but  they  rarely  win  the  competition  for  time  among  the  abundance  of  pressing 
requirements.  By  necessity  even  scheduled  training  must  sometimes  be  cancelled  or  modified  to 
meet  unexpected,  urgent  priorities.  Unit  leaders  focus  on  accomplishing  the  mission,  be  it  reset 
or  deployment — to  the  exclusion  of  intangible,  long-range  benefits  of  leadership  development. 
Another  aspect  of  the  current  reality  is  that  the  leadership  demands  on  junior  officers  have 
increased,  while  more  rapid  promotion  rates  have  decreased  their  opportunities  to  prepare 
thoroughly  for  the  increased  leadership  responsibilities.  On  one  hand  the  COE  demands 
leadership  development;  on  the  other  hand  the  OPTEMPO  precludes  it. 

Research  versus  Operational  Requirements 

Research  organizations  such  as  ARI  have  a  responsibility  to  develop  concepts,  strategies, 
methods,  technologies,  and  tools  that  further  the  Army’s  leader  development  goals  (U.S. 
Department  of  the  Army,  2007).  This  includes  evaluating  concepts,  technologies,  prototype  tools 
and  the  like  with  Soldiers  involved.  However,  the  OPTEMPO  and  deployment  pressures  make  it 
very  difficult  for  tactical  units  to  support  evaluation  activities.  This  sets  up  a  “catch  22” 
circumstance  where  the  inability  to  test  and  validate  new  leader  development  capabilities  may 
deprive  Soldiers  of  much-needed  tools.  Successfully  recruiting  units  or  Soldiers  does  not  ensure 
they  will  be  able  to  participate  fully  in  research  efforts.  Euture  research  teams  should  weigh  the 
troop  support  requirements  early  on  and  lay  out  vigorous  steps  to  secure  an  adequate  plan  for 
participation  of  Soldiers.  This  may  necessitate  enlisting  a  high-level  sponsor  and  convincing  the 
appropriate  chain  of  command  to  assign  a  high  priority  to  the  research  effort.  Emphasis  on 
utilizing  mechanisms  such  as  General  Officer  Steering  Committees,  Councils  of  Colonels,  and 
integrated  teams  of  stakeholder  representatives  may  be  worth  considering. 

Program  of  Instruction 


Suitability  of  Online  Delivery 

The  present  findings  raise  issues  regarding  the  suitability  of  distributed  learning  methods 
for  leader  self-development  programs.  Soldiers’  frequent  skepticism  about  online  leadership 
training  methods  indicates  that  there  are  disadvantages  to  be  weighed.  Program  architects  and 
designers  should  consider  potential  disadvantages  as  they  create  the  program  blueprint  and  make 
key  decisions  about  training  methods.  Beyond  the  technical  and  feasibility  issues  regarding 
online  training,  three  subtle  aspects  deserve  attention.  Eirst,  the  distributed  learning  option, 
because  of  its  visibility  and  resource  advantages,  may  lead  decision  makers  and  program 
executives  to  downplay  the  potential  role  of  alternative  training  methods.  Second,  the  suitability 
of  computer-enabled  methods  for  training  particular  leadership  skills  is  better  verified  during 
front-end  analysis  (EEA)  than  during  evaluation  of  prototype  tools.  Third,  the  culture  among 
tactical  units  is  predisposed  toward  face-to-face  methods  for  developing  leadership  skills,  and 
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new  distributed  learning  programs  face  the  challenge  of  persuading  Soldiers  to  accept  methods 
that  may  lack  traditional  interpersonal  dimensions. 

Importance  of  Interpersonal  Interaction 

Some  demonstration  participants  questioned  ALD’s  online  tools  on  the  yet- to-be-tested 
belief  that  leadership  development  requires  interaction  with  leaders/mentors  and  discussion  of 
leadership  lessons  with  peers.  This  was  a  major  reason  for  their  judgment  that  the  best  use  for  the 
ALD  toolkit  lies  in  institutional  training,  where  ideas  could  be  discussed  and  lessons  exchanged. 
It  may  be  feasible  in  the  future  to  create  online  leader  development  tools  for  collective  training 
applications  within  units.  Similarly,  incorporating  interpersonal  interaction  into  individual  Web- 
delivered  leader  development  tools  should  receive  careful  consideration.  This  could  be 
accomplished  by  remote  interaction  with  mentors  (synchronous  or  asynchronous),  student 
discussion  groups  (already  part  of  AXL  training),  computerized  interactive  discourse  (as  in  the 
AXL  modules),  participating  in  communities  of  practice  or  communities  of  interest  (such  as 
AKO  forums  or  Battle  Command  Knowledge  System  communities  of  practice),  feedback 
originating  from  experts  (as  in  the  TKML  instrument),  and  perhaps  other  techniques.  For  online 
self-development  training  to  be  accepted  and  used,  concerns  about  the  sterile  environment  of 
Web-delivered  training  must  be  addressed.  Both  the  users  and  the  chain  of  command  must  be 
convinced  of  the  effectiveness  of  online  training  or  they  will  ignore  the  tools. 

Quality  of  Training  Tools 

For  some  of  the  participants  in  the  ALD  demonstration,  aversion  to  Web-delivered 
training  stemmed  in  part  from  past  experiences  with  “check  the  block”  or  ineffective  training. 
With  the  high  OPTEMPO  demands,  there  is  no  tolerance  for  less  than  excellent  Web-delivered 
training.  Company  grade  officers  will  spend  neither  valuable  duty  time  nor  limited  Family  time 
on  training  unless  it  is  essential  and  excellent.  There  are  some  clear  lessons  on  what  Soldiers 
expect  in  Web-delivered  training.  Further  research  may  be  warranted  in  this  area,  but  training 
developers  should  consider  applying  these  lessons  in  the  design  of  training  for  junior  Army 
leaders.  The  training  must: 

•  Be  designed  specifically  for  the  Army  audience,  and  not  perceived  as  an  adaptation  of 
training  intended  for  some  other  audience. 

•  Apply  an  understanding  of  the  operational  Army  in  all  aspects — language, 
terminology,  scenarios,  instructional  content,  context,  etc. 

•  Organize  and  present  information  so  it  is  clear,  concise,  to  the  point,  easy  to  read,  easy 
to  follow,  and  free  of  “fluff  ” 

•  Build  or  enhance  skills  that  are  required  to  improve  immediate  performance  or  meet 
deployment  needs. 

•  Respect  the  limited  time  Soldiers  have  available  by  reducing  the  training  time  to  the 
absolute  minimum. 

•  Incorporate  an  automated  mechanism  that  enables  students  to  readily  track  their 
progress. 

•  Include  a  robust,  automated  book-marking  mechanism  so  students  can  break  in  the 
middle  of  a  module  and  later  return  effortlessly. 
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•  Provide  constructive  interaction  with  mentors  or  proxies  (e.g.,  artificially  intelligent 
tutors)  to  facilitate  learning  and  give  students  access  to  expert  viewpoints. 

•  Deliver  relevant,  useful,  timely  feedback  that  enhances  the  learning  process  and  helps 
the  student  understand  his/her  standing  or  progress. 

•  Present  a  style  and  tone  that  is  comparable  to  sophisticated  Web  sites  and  games  with 
which  the  target  audience  is  likely  to  be  familiar. 

Independence  of  Tools 

Originating  as  independent  components,  the  ALD  tools  were  not  designed  to  share 
common  themes,  principles,  scenarios,  etc.  Nor  were  they  designed  with  interdependencies  in 
mind,  such  as  relying  on  the  learning  in  one  module  to  feed  another  module  or  integrating  the 
cumulative  knowledge  and  skills  into  a  common  learning  forum.  In  the  ALD  operational 
demonstration,  no  unit  Soldiers  commented  on  the  lack  of  common  threads  across  the  various 
tools,  although  they  were  not  asked  directly  to  consider  the  issue.  In  retrospect,  the  program  may 
have  benefited  from  designing  a  family  of  related  tools  and  applying  principles  such  as  recurrent 
themes,  mutual  reinforcement,  successive  integration,  nesting,  scenario  commonality,  and 
capstone  assessment  to  their  development.  In  future  research,  it  would  be  worth  examining  the 
utility  and  value  of  creating  a  family  of  inter-related  leader  development  tools. 

Learning  Conditions 


Basic  Learning  Environment 

The  demonstration  participants’  comments  about  the  advantages  of  institutional  settings 
for  ALD  training  are  instructive.  They  considered  several  factors  important  to  define  a  favorable 
environment  for  leader  development  learning.  Their  comments,  along  with  the  insights  of  the 
research  investigators,  highlight  several  enabling  factors: 

•  Relatively  high  priority  on  developing  individual  leadership  skills. 

•  Availability  of  sufficient  time  for  learning  new  knowledge  and  skills. 

•  Presence  of  experts  available  to  spend  time  as  mentors  and/or  coaches. 

•  Availability  of  time  to  reflect  on  one’s  own  leader  development  strategy. 

•  Frequent  opportunities  to  meet  in  groups  to  share  experiences,  ideas,  and  the  like. 

•  Low  level  of  operational  demands  competing  for  time  and  energies. 

In  the  context  of  these  considerations,  the  institutional  environment  appears  to  be  ideal  for  leader 
development  training,  including  self-development  efforts.  However,  the  factors  listed  above  can 
serve  as  guidelines  for  units  trying  to  improve  their  own  leader  development  programs,  where 
efforts  could  benefit  from  increasing  the  emphasis  on  at  least  some  of  the  enablers.  The  factors 
may  also  help  program  architects,  designers,  and  developers  to  define  realistic  conditions  for  a 
successful  leader  development  program. 

Ready  Access  to  Learning  Tools 

Effective  use  of  Web-delivered  training  requires  that  Soldiers  have  adequate  access  to 
computers  with  internet  connections.  Although  computers  are  pervasive  in  many  environments, 
this  is  not  the  case  in  operational  units.  The  duties  of  most  junior  officers  and  NCOs  do  not 
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require  them  to  use  eomputers  with  internet  aeeess,  limiting  the  number  of  eomputers  assigned  to 
units.  Further,  eomputers  in  units  are  dedieated  to  speeifie  funetions,  sueh  as  eommand  and  staff 
operations.  While  unit  or  installation  learning  eenters  offer  computer  access,  the  extra  time 
needed  for  Soldiers  to  reach  them  becomes  an  impediment.  Expecting  Soldiers  to  use  their  own 
computers  for  training  may  be  unreasonable  from  two  perspectives.  It  involves  using  personal 
property  for  government  business,  and  it  infringes  on  a  Soldier’s  limited  Family  time.  The  latter 
is  especially  onerous  considering  that  Soldiers  at  home  station  are  either  returning  from  or 
preparing  for  a  deployment  away  from  their  families.  For  these  reasons,  a  training  program  FEA 
should  survey/analyze  the  expected  unit  infrastructure  to  determine  the  computer  availability  and 
network  assets  likely  to  exist  at  the  time  of  fielding.  If  needed,  computers  could  be  provided  to 
units  (and  left)  as  an  incentive  to  participate  in  Web-delivered  training. 

Soldier  Control  of  Learning  Conditions 

The  degree  to  which  a  learner  controls  his/her  progression  through  an  online  course 
should  be  considered  during  program  design.  Important  control  dimensions  include  pacing, 
distribution  of  effort,  break  points,  and  sequencing.  For  Army  leaders,  self-control  of  pacing  and 
distribution  is  a  must.  They  need  wide  flexibility  in  when  and  how  long  they  work  on  a  module, 
how  much  time  passes  between  sessions,  and  how  many  sessions  they  spend  on  a  module.  As 
discussed  earlier,  they  must  be  able  to  break  at  almost  any  point  in  a  module  and  return  later  to 
the  exact  break  point  (via  book-marking).  A  lockstep  or  prescribed  sequence  of  lessons  or 
modules  is  sometimes  imperative,  as  when  training  involves  sequential  dependencies.  However, 
Army  leaders  who  routinely  exercise  considerable  responsibility  may  not  favor  rigid  sequencing, 
and  some  unit  participants  expressed  dislike  for  the  rigidly  enforced  sequence  of  the  AFD  Web 
site.  There  are,  of  course,  inherent  risks  associated  with  giving  students  unfettered  latitude  in 
sequencing  or  progressing  through  a  training  program.  The  program  architects  should  identify 
and  weigh  the  risks  as  they  develop  the  training  strategy.  Consistent  with  other  training  design 
factors,  maximum  emphasis  should  be  placed  on  student  control  of  learning  conditions  for  Web- 
delivered  self-development  training  of  Army  leaders,  regardless  of  their  grade. 

Interaction  as  a  Part  of  the  Environment 

As  discussed  earlier.  Soldiers  tend  to  believe  that  leadership  development  must  involve 
the  individual’s  interaction  with  leaders,  mentors,  or  instructors  to  be  most  effective.  In  the  case 
of  the  AFD  toolkit,  many  demonstration  participants  felt  that  the  general  lack  of  interpersonal 
interaction  was  a  serious  shortcoming.  The  ideal  from  the  students’  perspective  is  face-to-face 
interaction  with  a  mentor/instructor.  Online  tools  can  leverage  distributed  learning  techniques  to 
achieve  interactivity.  The  training  development  question  becomes:  how  can  students  obtain 
advice  from  a  trusted  source,  in  a  context  that  is  relevant  to  their  situation,  when  they  need  it? 
Earlier  discussion  mentioned  several  online  interactive  techniques — remote  interaction  with 
mentors,  student  discussion  groups  (in  person  or  online),  communities  of  practice,  computerized 
interactive  discourse,  and  feedback  originating  from  experts  (synchronous  or  asynchronous). 
More  elaborate  techniques  include  intelligent  tutors  (computerized  agents)  and  electronic 
performance  support  systems.  Training  developers  can  determine  what  level  of  interactivity  is 
warranted,  using  cost-benefit  analysis  with  Soldiers  in  the  loop.  Training  researchers  should 
explore  ways  to  improve  the  interactive  dimensions  of  the  learning  environment.  They  should 
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also  explore  ways  to  ensure  the  interaction  is  perceived  as  credible.  Further  research  appears  to 
be  essential  if  Soldiers  are  to  be  convinced  that  online  training  is  a  valuable  way  to  build 
leadership  skills  and  knowledge. 

Time  Considerations 

Given  the  extremely  limited  time  Soldiers  in  operational  units  have  available  for  self¬ 
development  training,  how  much  time  is  likely  to  be  acceptable  for  completing  a  particular 
module?  The  answer  lies  in  a  cost-benefit  paradigm:  Soldiers  tend  to  weigh  the  time  required 
against  the  perceived  value  or  payoff  from  the  training.  The  greater  the  expected  payoff,  the 
greater  the  time  they  are  willing  to  invest.  Training  development  teams  must  keep  in  mind  that 
Soldiers  will  decide  the  value  and  acceptable  time  for  the  training — not  program  architects  or 
senior  Army  leadership.  The  FEA  should  use  objective  means  to  determine  what  leadership  skills 
the  leaders  themselves  recognize  as  critical  to  their  unit’s  mission  effectiveness.  The  FEA  should 
also  gauge  multiple  aspects  of  time:  how  much  total  time  is  acceptable  for  a  module  or  subject, 
how  much  time  can  be  spent  in  a  single  session,  and  how  many  calendar  days  is  the  module 
likely  to  span?  The  critical  EEA  issue  is  how  much  time  units  and  Soldiers  can  allocate  for  the 
training,  not  how  much  time  the  training  developer  thinks  should  be  allocated  to  fit  the  topic.  In 
addition,  the  AED  operational  demonstration  reinforced  the  importance  of  giving  students  an 
estimate  of  the  time  required  to  complete  each  module,  so  they  can  plan  their  time  strategy 
realistically. 


Fielding  and  Implementation 


Setting  Conditions  for  Success 

Eielding  and  implementing  new  programs,  including  Web-delivered  training,  require 
buy-in  by  the  targeted  units  and  their  leaders.  No  time  and  effort  will  be  invested  by  the  units 
without  buy-in  and  endorsement  on  the  part  of  commanders.  The  proponent  and/or  fielding 
agency  for  a  new  program  must  collaborate  with  commanders  to  determine  what  implementation 
conditions  are  feasible  for  the  targeted  units.  This  aspect  should  become  an  integral  part  of  the 
fielding  plan.  Another  key  goal  is  to  embed  the  new  capabilities  in  unit  training  programs,  and 
this  too  should  be  part  of  the  fielding  plan.  At  the  same  time,  the  proponent  or  fielding  agency 
should  remain  cognizant  of  the  constraints  with  which  commanders  must  live.  A  committee  of 
stakeholders,  perhaps  in  the  form  of  a  steering  committee,  may  be  able  to  facilitate  program 
fielding  and  implementation.  Although  this  may  require  significant  effort,  it  can  increase  the 
chances  for  success.  A  stakeholder/steering  committee  can  bring  a  better  appreciation  for  the 
situation  in  the  field  and  the  challenges  that  must  be  addressed.  The  committee  can  also  solicit 
the  senior  leadership  emphasis  needed  to  obtain  the  resources  and  priority  of  effort  required  to 
realize  the  benefits  of  the  new  program. 

Dedicating  Time  for  Leadership  Development 

As  the  saying  goes,  “There’s  no  such  thing  as  a  free  lunch.”  Trumpeting  a  new  program 
as  it’s  fielded  will  fall  on  deaf  ears  unless  commanders  decide  to  commit  unit  resources  to 
implementing  the  program.  In  the  case  of  leader  development  programs,  time  is  the  chief 
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resource  to  be  committed.  The  high  OPTEMPO  plus  Family  needs  make  time  an  ultra-precious 
resource  in  operational  units.  Expecting  leaders  in  tactical  units  to  spend  their  own  time  on  self¬ 
development  activities  is  unrealistic  in  the  current  COE  and  invites  program  dismissal.  Many 
Soldiers  already  receive  deployment-driven  tasks  to  accomplish  during  off-duty  hours,  eroding 
their  personal  and  Family  time.  Placing  AED  or  other  leadership  development  events  on  the  unit 
training  calendar  is  imperative  if  the  target  audience  is  to  realize  any  benefits.  The  Army’s  senior 
leaders  must  emphasize  leader  development  as  essential  preparation  for  deployment, 
acknowledging  that  leadership  skills  affect  mission  success  and  Soldier  survivability.  Unit 
leaders  can  translate  the  senior  leaders’  emphasis  into  allocating  and  protecting  sufficient  time 
for  leadership  development. 

Informing  and  Motivating  the  Target  Audience 

Soldiers  will  invest  their  time  and  effort  in  training  that  they  are  convinced  will  benefit 
their  duty  performance  and  mission  success.  Therefore,  the  target  audience  and  their  leaders 
must  be  armed  with  program  information  that  engenders  awareness  and  motivation.  More  than 
knowing  the  objectives  of  the  training,  they  must  know  how  the  new  training  will  benefit  them — 
especially  by  improving  performance  in  the  near  term.  Because  it  is  unsafe  to  assume  that  the 
target  audience  will  recognize  their  need  for  the  training,  the  AED  program  materials  contain 
motivational  information.  For  example,  the  TSP  includes  a  “benefits”  section  and  the  Web  site’s 
main  menu  includes  motivational  “tags”  for  each  AED  tool’s  link.  The  demonstration  results  do 
not  reveal  whether  the  toolkit  goes  far  enough  on  this  count.  Considering  the  time  constraints 
facing  unit  leaders,  future  development  and  implementation  teams  should  pay  special  attention  to 
convincing  the  target  audience  of  a  new  program’s  value.  The  supporting  materials  (in-briefing 
package,  student  guide,  posters,  etc.)  should  describe  succinctly  how  Soldiers  will  become  better 
leaders  and  how  unit  readiness  will  benefit.  In  addition  to  the  stakeholders/steering  committee 
option  mentioned  above,  working  through  the  AKO  community  may  be  useful  for  informing  and 
motivating  the  target  audience. 


Conclusions  and  Recommendations 

The  AED  toolkit  offers  innovative,  computer-based  technologies  for  enhancing  the 
Army’s  leadership  development  efforts.  As  the  culminating  phase  of  the  four-year  ATO 
program,  the  current  research  aimed  to  conduct  an  integrated  technology  demonstration  and 
assess  various  aspects  of  the  AED  program. 

The  toolkit  exploits  a  variety  of  distributed  learning  techniques.  The  Web-accessed  tools 
aim  to  improve  leadership  skills  by  helping  officers  to  become  self-aware,  adaptive  decision 
makers.  The  toolkit  contains  the  following  components: 

•  Diagnostic  instruments  to  enhance  self-awareness  of  strengths  and  weaknesses. 

•  Self-training  modules  to  build  metacognitive  skills  for  self-development. 

•  Critical  thinking  modules  to  train  structured  problem-solving  techniques. 

•  Film-based  case  study  vignettes  for  group  training  of  interpersonal  skills. 
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The  results  of  the  research  extend  the  knowledge  base  regarding  leader  development 
approaches  and  methods.  The  findings  are  of  primary  value  to  decision  makers,  program 
architects,  training  developers,  training  researchers,  and  program  evaluators.  The  lessons  learned 
provide  a  foundation  for  shaping  and  developing  future  leader  development  tools.  Additional 
findings  and  insights  on  training  technology  aspects  and  the  ALD  Web  site  can  be  found  in 
Woller  et  al.  (in  preparation). 


Conclusions 

The  ALD  tools  provide  a  reasonable  mix  of  diagnostic  and  training  capabilities,  but  the 
assessment  did  not  gauge  completeness  of  the  toolkit.  The  tools  appear  to  be  more  suitable  for 
use  in  pre-commissioning  programs  and  Army  schools,  rather  than  tactical  units.  The  suitability 
of  the  training  model  remains  an  open  question.  Additional  research  is  needed  to  determine 
program  completeness,  optimal  utilization  model,  and  tailoring  required  for  institutional  settings. 

On  the  whole,  the  demonstration  participants  expressed  moderately  positive  regard  for 
most  aspects  of  the  ALD  toolkit,  including  the  acceptability  of  the  various  modules  and  the 
learning  effectiveness  and  value  of  using  them.  At  the  same  time,  there  is  substantial  room  for 
improvement.  A  handful  of  general  recommendations  emerged,  based  on  Soldiers’  expectations 
of  online  training  packages: 

•  Make  the  tools  more  Soldier  friendly,  more  concise,  and  less  time  consuming. 

•  Incorporate  doctrinal  references  or  other  well  known  sources  of  key  information. 

•  Store  student-entered  responses  for  later  retrieval  or  presentation  (LASTS). 

•  Automate  interactive  capabilities  to  orchestrate  forward-feed,  cumulative  progression. 

•  Enable  breaking  anytime  with  assured  saving  and  later  retrieval  of  information. 

•  Update  the  AXL  videos  to  reflect  the  current  MDMP  process  used  in  the  COE. 

•  Enhance  the  CTS  modules  by  incorporating  more  realistic  conditions  and  exercises. 

At  least  in  tactical  units.  Soldiers  may  view  distributed  learning  methods  as  marginally 
suitable  for  leader  self-development  purposes.  The  potential  disadvantages  of  online  tools  must 
be  considered  carefully.  An  important  dimension  is  interpersonal  interaction  with  leaders, 
mentors,  and  peers.  The  interactive  features  of  the  AED  toolkit  were  useful  but  did  not  go  far 
enough.  Research  would  be  needed  to  explore  techniques  for  improving  the  interpersonal 
dimensions  of  online  tools. 

The  COE  has  a  powerful  impact  on  the  professional  development  of  junior  leaders,  both 
officers  and  NCOs,  in  tactical  units.  Eour  aspects  of  the  COE  stand  out:  (a)  rapid  promotion  of 
junior  leaders,  (b)  OPTEMPO-driven  scarcity  of  time  for  leadership  development,  (c)  need  for 
more  leader  development  opportunities,  and  (d)  imperative  to  include  self-development  tasks  in 
the  unit  training  schedule. 

Several  factors  define  a  favorable  environment  for  leader  development  learning.  These 
factors  include  (a)  high  priority  on  individual  leadership  training,  (b)  adequate  time  for  learning 
and  reflection,  (c)  availability  of  mentors/coaches,  and  (d)  opportunities  for  group  interaction. 
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The  current  information  technology  infrastructure  in  tactical  units  does  not  support  wide- 
scale  distributed  learning.  General  purpose  computers  are  relatively  scarce,  especially  below 
battalion  echelon.  Learning  centers  with  computers  typically  impose  a  tax  in  the  form  of  travel 
time  and  inconvenience.  Expecting  Soldiers  to  use  their  own  computers  for  self-development  is 
questionable  because  it  involves  using  personal  property  for  government  business  and  it  infringes 
on  a  Soldier’s  personal  and  Family  time. 

The  limitations  of  the  current  effort  point  to  follow-on  research  in  several  areas.  A  trial 
implementation  with  RC  units  would  yield  valuable  feedback  and  lessons  for  crucial  elements  of 
today’s  modular  force.  Extending  the  database  to  institutions  and  additional  AC  units  would 
generate  broader  conclusions  and  bring  greater  focus  to  future  efforts  to  develop  leader  self¬ 
development  programs.  Insights  on  an  effective  self-development  training  strategy  would  help 
guide  program  architecture  and  design.  And  further  research  on  innovative  online  methods  for 
leader  self-development,  including  best  practices  for  interpersonal  interaction,  would  offer  useful 
knowledge  and  guidelines. 


Recommendations 

This  report  has  presented  the  findings  of  an  operational  demonstration  and  assessment  of 
the  ALD  program  for  self-development  of  Army  leaders.  By  harnessing  the  findings,  program 
architects  and  training  developers  can  refocus  Army  efforts  to  enhance  the  process  for  producing 
competent,  self-aware  and  adaptive  leaders.  The  authors  offer  the  following  recommendations  to 
boost  the  Army’s  efforts: 

•  Consider  improving  the  ALD  toolkit  by  incorporating  recommended  modifications, 
perhaps  as  a  near-term  programmatic  step. 

•  Target  timely  expansion  of  the  database  on  ALD  training  effectiveness  and  impact,  to 
include  trial  implementations  with  institutions,  RC  units,  and  additional  AC  units. 

•  Pursue  follow-on  research  on  innovative  distributed  learning  methodology  for 
delivering  Soldier  friendly,  high-payoff  leader  development  training. 

•  Outline  contingency  plans  for  follow-up  evaluation  in  case  the  ALD  program  becomes 
available  at-large. 

The  progressive  research  culminating  in  the  operational  demonstration  and  assessment  of 
the  ALD  toolkit  sets  the  stage  for  enhancing  the  effectiveness  and  efficiency  of  Army  leadership 
development  programs.  By  helping  units  and  institutions  build  highly  competent  leaders  and 
leader  teams  across  the  force,  the  research  is  playing  a  key  role  in  Army  transformation.  As 
transformation  and  deployment  requirements  continue  to  expand  the  need  for  excellence  in  Army 
leaders,  the  products  and  findings  of  the  cumulative  research  can  help  build  leader  competencies 
that  are  critical  enablers  of  full  spectrum  dominance. 
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AREORGEN 
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BOLC 

CCC 
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Application  Exercise 

Army  Knowledge  Online 

Accelerating  Eeader  Development 

Advanced  Non-Commissioned  Officer  Course 

Army  Force  Generation 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

Army  Technology  Objective 

Army  Excellence  in  Eeadership 

Brigade  Combat  Team 

Basic  Non-Commissioned  Officer  Course 
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Captains  Career  Course 

Command  and  General  Staff  College 

contemporary  operational  environment 

Critical  Thinking  Skills 

Effective  Self- Appraisal 
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hypertext  markup  language 

Eeader  Adaptability  Self-Training  System 

learning  management  system 

mean 

median 

Multifactor  Eeadership  Questionnaire 
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operational  tempo 

political,  military,  economic,  social,  information,  and  infrastructure 

Reserve  Component 

Reserve  Officer  Training  Corps 

Self  Regulation 

Self  and  Social  Awareness 

standard  deviation 

Tacit  Knowledge  for  Military  headers 
Training  Support  Package 
Unit  Cohesion  Index 
United  States  Military  Academy 
executive  officer 
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